Over at Airbyte, we had a project this quarter to update all of our build & publish processes over to building multi-arch (AMD and ARM) docker images. As Airbyte runs entirely within docker, getting a smooth local experience for folks on M1/(2?) Macs was important. We had a long lived support thread (1) where you can see us grow through all the phases - from "nothing works", to "our deps don't work", to "the platform works" and finally to "the connectors work"!
Assuming your base images are themselves already multi-arch, most of the tooling we needed was already built into the `dockerx` build tool, which is awesome - check it out if you haven't (2). Docker has bundled all the tooling and emulation packages (qemu) needed into a single docker image that can publish multi-arch docker images for you! You run docker to emulate docker to publish docker... There are some interesting things that you'll need to do if you publish multi-stage builds, like publish a tmp tag and delete it when you are done, but it's not /too/ terrible. Since Airbyte is OSS, you can check out our connector publish script here (3) to see some examples.
I'd recommend that spending the time to work your multi-arch tooling - not only does it make the local dev experience faster/better, it:
1. unlocks ARM cloud compute, which can be faster/cheaper in many cases (AWS)
2. removes a class of emulation bugs when running AMD images on ARM - mostly around networking & timing (in the JAVA stack anyway)
Assuming your base images are themselves already multi-arch, most of the tooling we needed was already built into the `dockerx` build tool, which is awesome - check it out if you haven't (2). Docker has bundled all the tooling and emulation packages (qemu) needed into a single docker image that can publish multi-arch docker images for you! You run docker to emulate docker to publish docker... There are some interesting things that you'll need to do if you publish multi-stage builds, like publish a tmp tag and delete it when you are done, but it's not /too/ terrible. Since Airbyte is OSS, you can check out our connector publish script here (3) to see some examples.
I'd recommend that spending the time to work your multi-arch tooling - not only does it make the local dev experience faster/better, it:
1. unlocks ARM cloud compute, which can be faster/cheaper in many cases (AWS)
2. removes a class of emulation bugs when running AMD images on ARM - mostly around networking & timing (in the JAVA stack anyway)
Links:
1. https://github.com/airbytehq/airbyte/issues/2017
2. https://docs.docker.com/buildx/working-with-buildx
3. https://github.com/airbytehq/airbyte/blob/master/tools/integ...