Just to follow up on my own question (though I appreciate the siblings):
I found this article with some overview and example command lines: https://www.docker.com/blog/multi-arch-images/ . As best I can tell, you don't actually need a custom builder. You can just skip the `docker buildx create` and go straight to `docker buildx build` after one workaround below.
E.g. if I wanted to start building ARM binaries on a x86 host, is that the sort of thing this would enable?