This is only a problem if you're using Xcode-bundled Swift, right? It seems like you can easily run Swift on earlier version of macOS completely independently of Xcode if you want to.
Technically true, though if I’m going to give up the advantages of the standard toolchain and the ability to ship App Store apps, I’m not sure why I’d use Swift at all.
Some people like Swift on its own merits but I do not. It’s a terrible fit for the kinds of programming I want to do.