1. You can use the WebDriver protocol[0] without selenium
2. Firefox's WebDriver endpoint geckodriver[1] is a layer above Firefox/Gecko's native Marionette protocol[2]
So you doubly don't need to use selenium, you can write your own WebDriver client (which should be cross-browser assuming the browsers either support WebDriver natively or have a WD layer of some sort installed) or you can use raw Marionette (either a hand-rolled client or an existing client)
Patch author here: To add to the other comments, you could also use SlimerJS[1]. Though I don't know if I'd consider that lower level. We're open to supporting other ways of controlling FIrefox and have been gauging feedback on what to support.