I recently (yesterday) ran into emulation issues when I tried to run my dedicated game server build for Linux (that I produced from Unity) inside an amazon Linux docker container on my M2 Mac. I use docker for apple silicon that supposedly allows you to run amd64 images on arm (which M1/M2 macs are). Yes even specifying the correct —-platform and upgrading the emulation software.
It was only when I opened up all logging in a dev build did I discover there was some syscall from the mono side of things that wasn’t emulated properly enough. I thought it ironic recalling how many people had sold containers as lighter weight VMs that remove “works on my machine” hassles. Anyways, I just spun up an AWS EC2 instance to test this nothing part of the process.
I know podman uses qemu and maybe it could run this Unity based workload and maybe I could uninstall docker but eesh the boring image tech is starting to become a bit of a hassle. I’ll probably just keep using EC2 instances or write up a vagrant file to create a local VM. I’m sure eventually my workload will be fully supported eventually so I can optionally just wait.
Containers can and do remove the "it works on my machine" problem when you're on the same platform, and preferably the same OS exactly. The big deal here at least in my experience is the ability to completely replicate your environment, however you still need the underlying OS, because docker is not a VM.
It sounds to me like you're running a container on a massively different architecture and I think it's still pretty cool that it runs at all. Obviously when trying to run a game, which needs access to hardware and system calls beyond the use case of a simple CLI tool, running the container on a different architecture becomes more of a long shot.
I think given the circumstances it's actually kind of amazing you got this far, and if you're able to go all the way with your game with emulation, that'd be pretty cool.
And on M1/M2 Macs that means an arm64 Linux VM. So whatever issue came up here probably would come up attempting to run amd64 containers on regular arm64 linux (assuming the right compatibility tooling is installed).
> I thought it ironic recalling how many people had sold containers as lighter weight VMs that remove “works on my machine” hassles.
Containers aren't VMs, they're Linux namespaces. If you want to run a container you need Linux. If you want to run Linux on MacOS you need virtualization. There's no "lightweight" VM about it - it's still a VM.
It’s worth noting there are different kinds of containers out there, and Docker is just one “ecosystem”. Docker on Windows and macOS runs in a VM, which is kind of a bummer because it means you can’t really allocate all your RAM for containers. By default Docker likes to max out at half the system RAM, which of course you can change post-install, but I feel like there’s a trade off or disadvantage in such a scenario vs Docker (or LXC or podman or something else) running on Linux.
It might be that the RAM is reserved up-front on Windows and macOS but not on Linux? I’m not sure.
On Windows and macOS Docker runs Linux in a VM and then all the containers attach to that VM. I think this has changed to WSL2 on Windows of late but it's still the case on macOS AFAIK.
Windows actually has some kind of native containerization but I believe it's based on virtualization and not an equivalent of namespaces on Linux. I'm not even sure if such a feature would make sense on Windows in the same way it does on Linux.
But that's my point. It seems from this thread people don't understand what docker is except that they can kinda use it like a VM. But it's fundamentally different.
WSL2 is a Linux VM (albeit with some advanced features such as dynamic memory allocations, GPU & file sharing, etc.) on Windows hypervisor, and Docker for Windows just uses their own WSL2 VM.
It also 'magically' exposes Docker socket to your own WSL2 instances/VMs.
No, they are not. Containers are namespaced process groups (CGroups) with a sandboxed filesystem and (optionally) sandboxed network layer via bridge interfaces and virtual devices.
It is, and Docker for Mac uses QEMU for `--platform`, just like Podman AFAIK. The issue here is that it's QEMU user-space emulation, which does not support some syscalls, like `ioctl`.
The fix here would be to spin an amd64 VM and run the binary or install Docker there, instead of relying on the more convenient - but less compatible - user-space emulation of Docker. Not very related to this very interesting presentation on virtualizing ARM macOS guests on QEMU hosts..
Only on older setups and on the default config. Updated MacOS (13+) and updated docker provides an option to use Rosetta and the new virtualization system. That, I believe, replaces QEMU for x86 emulation with the Apple emulator.
I've had substantial success with that method; not only does it seem a bit faster, but some containerized apps that very frequently failed to start (e.g. Apache Pulsar) work consistently in that mode. I suspect this difference in experience speaks to bugs in QEMU that trigger in the presence of certain application behavior (in Pulsar's case, something weird happening with the JVM on startup).
QEMU's still an incredible feat of engineering and something I frequently use on Linux, though. It just seems that the Apple emulation on Macs is better (which makes sense, given that it's built by the people who made the semi-proprietary silicon it's emulating from).
Can I ask a question? It seems like your post is combining two separate things. As I understand it and based on earlier comments in the thread:
1. To run a Linux container you need to be running Linux
2. This is why Docker for Max has always run a VM under the covers. It uses qemu.
3. This was true even before Apple Silicon when everything was x86_64 and nothing was emulated
Now you are bringing up Rosetta and x86 emulation. While I understand sometimes it’s necessary to run a container that hasn’t been compiled for aarch64 (or whatever they call Apple) so I can see why the actual cpu emulation capabilities of qemu or Rosetta would be necessary there, this seems to be beside the point.
if you have no Linux system involved because you have no VM, how would that work? What would the base system “under” the container be and what would respond to Linux’s OS-level calls?
> sometimes it’s necessary to run a container that hasn’t been compiled for aarch64
I have several containers in that boat, for which no ARM versions exist at all. Those containers were previously using QEMU to emulate x86 on ARM linux (in Docker's VM on an ARM Mac). That emulation encountered the failures I described, and was slow.
Something I don't know and am curious about is how Docker-for-Mac is actually using Rosetta 2. Is it running an additional Linux VM containing an x86 Linux OS, and running that VM through a Rosetta-2-enabled hypervisor? Or is Rosetta 2 distributed as a Linux program that is being invoked inside Docker's pre-existing aarch64 Linux VM instead of (or inside of?) QEMU?
Edit: as for your question:
> if you have no Linux system involved because you have no VM, how would that work?
Yes, Rosetta 2 is being used by Docker’s pre-existing aarch64 Linux VM. So it’s only translating userspace x86 code to Arm, then running it on the ARM Linux VM. Syscalls, etc. are still handled by the (aarch64) Linux VM.
> I thought it ironic recalling how many people had sold containers as lighter weight VMs
That's your problem there, containers are by design not VMs. Anyone who sold them as such aren't understanding the tech and you shouldn't have blindly trusted them.
Even when the arch matched, macOS was a bad platform for running containers. I'm not even sure you can call them "edge cases" because it was basic things like volumes not working right.
macOS is engineered for users and desktops, not servers and development. Apple's long history of blocking virtualizing macOS is a testament to this.
MacOS has been virtualizable in x86 since mac-on-x86 was released.
It also is engineered for development. Maybe not your particular type of development. Apple silicon isn't technically development-unfriendly, it's just apple silicon makes it less easy to do x86 development. But so does mips, sparc, ppc, and other architectures.
I'm using UTM (nice frontend to Qemu) and the performance is okay-ish on a base mbp m1. The desktop is fluid enough for basic usage. I use it mainly for remote access client apps that I don't want running on my OS, like Citrix.
UTM is fantastic for macOS-on-macOS emulation on Apple Silicon devices. Historically, macOS VMs have been so slow on every other virtualization platform. UTM uses Apple's native virtualization framework and it's a godsend for testing older or newer OS's.
I used parallels in the past. Mainly to spin up macOS VMs for setup testing with ansible (I maintain a macOS build farm for our studio) And I found the performance decent enough. It all tanks the moment more than one instance is running though.
what operating systems are you running on utm? I've tried it, and for some it's great, for others, usually one of the bsd's, there's always some small problem. And the gui feels kind of incomplete.
Weirdly, virtualbox on linux was usually my go to for trying out other operating systems. It also felt like the most straightforward to use.
VMware passes through the id of the physical hardware into the guest VM. So if you're running on apple hardware, this works. If you're running on a PC, there are hacks to 'borrow' the serial number of some ancient imac and have it work still (even ancient systems are still allowed to log into icloud, and those ancient systems didn't do any cryptography so all you need is a valid serial number that nobody else is using).
> If you're running on a PC, there are hacks to 'borrow' the serial number of some ancient imac and have it work still (even ancient systems are still allowed to log into icloud, and those ancient systems didn't do any cryptography so all you need is a valid serial number that nobody else is using).
Hackintosh users have to do something similar to be able to sign into iCloud.
The serial number doesn't have to be valid (as in existing on a physical machine) to work, though. It just needs to look valid (be generated using the same methodology as real serials). In fact in order to prevent accidentally using a serial tied to a machine owned by somebody else, the recommended procedure is to generate a serial and check its AppleCare status to verify that it's not tied to a real machine, and if it is to regenerate and check until you find one that isn't.
> even ancient systems are still allowed to log into icloud, and those ancient systems didn't do any cryptography so all you need is a valid serial number that nobody else is using
The ancient random serial numbers are also in a standard format.
> Apple devices manufactured after 2010 generally have 12-character alphanumeric serial numbers, with the first three digits representing the manufacturing location, the following two indicating the year and week of manufacture, the next three digits providing a unique identifier, and the last four digits representing the model number.
But if this serial number was shared between several people, all of whom might have logged in with a different apple cloud id, would that somehow cause apple to trigger something?
i could be completely wrong here, but i do seem to remember something about there being some physical chip in the machine that needed to be present for e.g. iMessage to work. this was a while ago but iirc they also somehow incorporated local Apple ID account management into that system.
Note that the EU's Digital Markets Act will probably stop all that bs, as requiring to use one core platform service to develop for, and distribute to, another (iOS) is against that law.
Third-party app stores and outright website-based app distribution are coming next year anyway.
That's interesting, I did not know about this legislation. I'm not European, but it seems that even outside of Europe, people will benefit from this change.
If Apple was compelled to support a first-class iOS development experience on Windows, all this would do is immovably stunt the continued improvement of iOS development on macOS. There is a material degree of OS integration / cooperation in IOS development with Xcode. If Apple released a second-class development experience I’ve no doubt that you’d spare no time in making similar insinuations that it’s some sort of artificial constraint.
Xcode and iOS Simulator on other platforms would involve porting AppKit as well as writing all-new Darwin virtualization for those platforms. Xcode is thoroughly a Mac native app, and the iOS/watchOS/tvOS/etc simulators currently just run the respective userlands on top of the underpinnings that macOS shares with other Apple platforms.
It'd be rather similar to the situation with Docker, where Xcode and the simulators would run markedly better on macOS than on other platforms due to fewer layers being necessary. Developing for iOS on Windows or Linux would technically be possible but it wouldn't be very pleasant.
Been using Xcode since 2012 and I think it's great. It has it's flaws, sure, but overall I think it's a really good development experience. What do you dislike about it?
I've been using it since it was known as Project Builder and also don't have many complaints. It's not perfect, but it's not bad. I spend plenty of time in the IntelliJ-based Android Studio and haven't found it to be much of an improvement (and in fact, find that its "smarts" get in the way as often as they help).
Well, QEMU has no graphical acceleration for macOS (no virtio drivers for XNU and Quartz) so it's basically unusable - software rendering has been historically borked on macOS, and never well tested.
I'm sure it's possible, depending on the kind of application and the tools you use to build it. But xcode won't build your application if you don't link your account. I believe it needs that to "sign" the application with apple.
There may be ways to get around this but I don't of any.
You can manage your signing identity entirely using Apple's website and just not use automatic provisioning. (I do that anyway even though I build on macOS as I just don't want our build system tied to Xcode just so we can use the automatic provisioning.)
It would be interesting for this audience to compare QEMU against VirtualBox..
My own bias is that I will be purchasing a Mac Studio around XMas as a flutter app build machine.
I know that VirtualBox has guest OS additions to pass through graphics acceleration from the Host OS to the Guest OS. Does QEMU have this for MS Windows and Linux *6 guest OSes?
As a personal VM machine, vbox is fine. But for anything more involved, it's got a lot of rough edges.
We used to use virtualbox at work to run our CI workload (spawning a fresh VM for every job). Long story short: It does not scale well at all. Its scripting API is terrible, we ran into endless network problems (for instance, its builtin dhcp server would sometimes fail to flush expired IPs, leading to running out of IPs to give), and it would sometimes just freeze up entirely, requiring us to restart the virtualbox services.
We switched over to kubevirt[0] (essentially qemu/kvm inside a kube cluster) to handle our CI, and it's been a much, much smoother sailing since then. And it even allowed us to run macos VMs on linux hosts, something that we never managed to get working in vbox.
> Does QEMU have this for MS Windows and Linux *6 guest OSes?
QEmu has virgl[1] for paravirtualized graphics, but its windows drivers are unsigned afaict, so you can't really use it there. Works fine on linux, so long as your mesa and kernel aren't super ancient.
The quick summary is VirtualBox vs kvm/qemu is a classic commercial product vs open source comparison. Without going too deep into the actual architecture, VirtualBox is much easier to get started with (although check out something like Proxmox), but qemu is much more powerful. AFAIK, there are no features in VirtualBox that can't also be done with kvm/qemu, but there's a lot of things that can be done with kvm/qemu that can't be done in VirtualBox.
I say kvm/qemu because the actual hypervisor is kvm, while qemu is used for control and device emulation. qemu can also emulate full machines, which is obviously much slower but can be extremely useful.
The command line strings for Qemu can be enormous and complicated because they have changed over time, whats shown on page49 is very simplified, but should get some people going.
Well, not just that, but Qemu isn't a platform virtualization utility like Hyper-V or Virtualbox. Qemu is an emulator of general machine platforms that happens to offer virtualization APIs for performance boosts in specific configurations (x86 on x86, most specifically).
This is what allows it to simultaneously pretend to be a reference PREP platform, G3 machine, G4 machine, ARM9TDMI mobile device, ARM-Cortex EFI platform, SPARC server or even some hodge-podge machine of mixed emulated components. People treat it as equivalent to VBox or VMware; but it's not, it's a different tool that happens to offer a similar overlap of use cases (like using the butt of a screwdriver as a hammer).
> Qemu isn't a platform virtualization utility like Hyper-V or Virtualbox
I know, thats why I could run arm based OS on those twos on an x86 or x64 cpu.
Theres a lot of work thats gone into Qemu to emulate the different processors, question is have they found undocumented features of cpu's when trying to emulate them?
Technically, a good chunk of the ARM boards that Qemu supports are completely undocumented. So yes?
The emudev scene is pretty close-knit, there are shared contributors amongst MESS, MAME, Qemu, etc that spread research advances. In addition, a lot of the breakthrough work in undocumented/edge features is done by decappers, hardware researchers and computer engineers. This research then distills down into the more advanced emulator platforms before working its way into commodity documentation and hobby emulators.
