I recently (yesterday) ran into emulation issues when I tried to run my dedicated game server build for Linux (that I produced from Unity) inside an amazon Linux docker container on my M2 Mac. I use docker for apple silicon that supposedly allows you to run amd64 images on arm (which M1/M2 macs are). Yes even specifying the correct —-platform and upgrading the emulation software.
It was only when I opened up all logging in a dev build did I discover there was some syscall from the mono side of things that wasn’t emulated properly enough. I thought it ironic recalling how many people had sold containers as lighter weight VMs that remove “works on my machine” hassles. Anyways, I just spun up an AWS EC2 instance to test this nothing part of the process.
I know podman uses qemu and maybe it could run this Unity based workload and maybe I could uninstall docker but eesh the boring image tech is starting to become a bit of a hassle. I’ll probably just keep using EC2 instances or write up a vagrant file to create a local VM. I’m sure eventually my workload will be fully supported eventually so I can optionally just wait.
Containers can and do remove the "it works on my machine" problem when you're on the same platform, and preferably the same OS exactly. The big deal here at least in my experience is the ability to completely replicate your environment, however you still need the underlying OS, because docker is not a VM.
It sounds to me like you're running a container on a massively different architecture and I think it's still pretty cool that it runs at all. Obviously when trying to run a game, which needs access to hardware and system calls beyond the use case of a simple CLI tool, running the container on a different architecture becomes more of a long shot.
I think given the circumstances it's actually kind of amazing you got this far, and if you're able to go all the way with your game with emulation, that'd be pretty cool.
And on M1/M2 Macs that means an arm64 Linux VM. So whatever issue came up here probably would come up attempting to run amd64 containers on regular arm64 linux (assuming the right compatibility tooling is installed).
> I thought it ironic recalling how many people had sold containers as lighter weight VMs that remove “works on my machine” hassles.
Containers aren't VMs, they're Linux namespaces. If you want to run a container you need Linux. If you want to run Linux on MacOS you need virtualization. There's no "lightweight" VM about it - it's still a VM.
It’s worth noting there are different kinds of containers out there, and Docker is just one “ecosystem”. Docker on Windows and macOS runs in a VM, which is kind of a bummer because it means you can’t really allocate all your RAM for containers. By default Docker likes to max out at half the system RAM, which of course you can change post-install, but I feel like there’s a trade off or disadvantage in such a scenario vs Docker (or LXC or podman or something else) running on Linux.
It might be that the RAM is reserved up-front on Windows and macOS but not on Linux? I’m not sure.
On Windows and macOS Docker runs Linux in a VM and then all the containers attach to that VM. I think this has changed to WSL2 on Windows of late but it's still the case on macOS AFAIK.
Windows actually has some kind of native containerization but I believe it's based on virtualization and not an equivalent of namespaces on Linux. I'm not even sure if such a feature would make sense on Windows in the same way it does on Linux.
But that's my point. It seems from this thread people don't understand what docker is except that they can kinda use it like a VM. But it's fundamentally different.
WSL2 is a Linux VM (albeit with some advanced features such as dynamic memory allocations, GPU & file sharing, etc.) on Windows hypervisor, and Docker for Windows just uses their own WSL2 VM.
It also 'magically' exposes Docker socket to your own WSL2 instances/VMs.
No, they are not. Containers are namespaced process groups (CGroups) with a sandboxed filesystem and (optionally) sandboxed network layer via bridge interfaces and virtual devices.
It is, and Docker for Mac uses QEMU for `--platform`, just like Podman AFAIK. The issue here is that it's QEMU user-space emulation, which does not support some syscalls, like `ioctl`.
The fix here would be to spin an amd64 VM and run the binary or install Docker there, instead of relying on the more convenient - but less compatible - user-space emulation of Docker. Not very related to this very interesting presentation on virtualizing ARM macOS guests on QEMU hosts..
Only on older setups and on the default config. Updated MacOS (13+) and updated docker provides an option to use Rosetta and the new virtualization system. That, I believe, replaces QEMU for x86 emulation with the Apple emulator.
I've had substantial success with that method; not only does it seem a bit faster, but some containerized apps that very frequently failed to start (e.g. Apache Pulsar) work consistently in that mode. I suspect this difference in experience speaks to bugs in QEMU that trigger in the presence of certain application behavior (in Pulsar's case, something weird happening with the JVM on startup).
QEMU's still an incredible feat of engineering and something I frequently use on Linux, though. It just seems that the Apple emulation on Macs is better (which makes sense, given that it's built by the people who made the semi-proprietary silicon it's emulating from).
Can I ask a question? It seems like your post is combining two separate things. As I understand it and based on earlier comments in the thread:
1. To run a Linux container you need to be running Linux
2. This is why Docker for Max has always run a VM under the covers. It uses qemu.
3. This was true even before Apple Silicon when everything was x86_64 and nothing was emulated
Now you are bringing up Rosetta and x86 emulation. While I understand sometimes it’s necessary to run a container that hasn’t been compiled for aarch64 (or whatever they call Apple) so I can see why the actual cpu emulation capabilities of qemu or Rosetta would be necessary there, this seems to be beside the point.
if you have no Linux system involved because you have no VM, how would that work? What would the base system “under” the container be and what would respond to Linux’s OS-level calls?
> sometimes it’s necessary to run a container that hasn’t been compiled for aarch64
I have several containers in that boat, for which no ARM versions exist at all. Those containers were previously using QEMU to emulate x86 on ARM linux (in Docker's VM on an ARM Mac). That emulation encountered the failures I described, and was slow.
Something I don't know and am curious about is how Docker-for-Mac is actually using Rosetta 2. Is it running an additional Linux VM containing an x86 Linux OS, and running that VM through a Rosetta-2-enabled hypervisor? Or is Rosetta 2 distributed as a Linux program that is being invoked inside Docker's pre-existing aarch64 Linux VM instead of (or inside of?) QEMU?
Edit: as for your question:
> if you have no Linux system involved because you have no VM, how would that work?
Yes, Rosetta 2 is being used by Docker’s pre-existing aarch64 Linux VM. So it’s only translating userspace x86 code to Arm, then running it on the ARM Linux VM. Syscalls, etc. are still handled by the (aarch64) Linux VM.
> I thought it ironic recalling how many people had sold containers as lighter weight VMs
That's your problem there, containers are by design not VMs. Anyone who sold them as such aren't understanding the tech and you shouldn't have blindly trusted them.
Even when the arch matched, macOS was a bad platform for running containers. I'm not even sure you can call them "edge cases" because it was basic things like volumes not working right.
macOS is engineered for users and desktops, not servers and development. Apple's long history of blocking virtualizing macOS is a testament to this.
MacOS has been virtualizable in x86 since mac-on-x86 was released.
It also is engineered for development. Maybe not your particular type of development. Apple silicon isn't technically development-unfriendly, it's just apple silicon makes it less easy to do x86 development. But so does mips, sparc, ppc, and other architectures.
It was only when I opened up all logging in a dev build did I discover there was some syscall from the mono side of things that wasn’t emulated properly enough. I thought it ironic recalling how many people had sold containers as lighter weight VMs that remove “works on my machine” hassles. Anyways, I just spun up an AWS EC2 instance to test this nothing part of the process.
I know podman uses qemu and maybe it could run this Unity based workload and maybe I could uninstall docker but eesh the boring image tech is starting to become a bit of a hassle. I’ll probably just keep using EC2 instances or write up a vagrant file to create a local VM. I’m sure eventually my workload will be fully supported eventually so I can optionally just wait.
I love and hate knowing so much.