This in combination with the 3D acceleration work going into QEMU [1] could end up being a pretty compelling solution for running the x86 Linux version of Steam on macOS.
I was under the impression that Box86 would not work on the M1 because it wants to run in 32 bit mode, and Apple did not implement 32 bit mode on their processors.
Does anyone understand (or have any theories on) how this actually works? I don't understand how it's possible. Surely they didn't write a Linux version of Rosetta, it must be talking to the host OS somehow—but how? Where is the boundary?
I wonder how they handled TSO mode. Do they enable it for the whole VM? Otherwise I cannot see how it would safely work given they could be context-switched anytime by the guest kernel.
It seems pretty clear from TFA - there’s a directory share with the host, and given that Rosetta isn’t an emulator, but rather a translation layer, they don’t need a Linux version: x86 instructions go in, arm64 come out.
But surely that would be too slow? Although Rosetta is great at caching instructions ahead of time, it does need to emulate a lot of code (ie, anything generated at runtime).
It’s doing an AOT translation of x86-64 opcodes to ARM64 equivalents. There isn’t really any back and forth, it just digests the binary all at once.
This would still be pretty slow (see: Microsoft’s version of this under Windows ARM) due to the need to issue a ton of memory fence instructions to make ARM’s looser memory model behave like Intel’s, except that Apple baked the ability to switch the CPU into an Intel-like memory model directly into the silicon.
> It’s doing an AOT translation of x86-64 opcodes to ARM64 equivalents. There isn’t really any back and forth, it just digests the binary all at once.
No it's not. Apple is not immune from fundamental computer science principles whatever their marketing team says, and even the original keynote acknowledged that Rosetta 2 emulates some instructions at runtime.
Imagine you're running Python under Rosetta. The original Python interpeter takes Python code, translates it into x86 assembly, and runs that x86 assembly. Those x86 instructions did not exist prior to execution! Even if Rosetta could translate the entire interpreter into ARM code, the interpreter would still be producing x86 assembly.
Other types of programs produce code at runtime as well. Rosetta 2 is able to cache a very impressive amount of instructions ahead of time, but it's still doing emulation.
Yes, it includes a runtime JIT component for apps that happen to dynamically generate x86-64, but in all other cases the binary is AOT translated. It does this by inserting (during the AOT translation) function calls to a linked in, in-process translation function whenever it sees mmap’d or malloc’d regions being marked for execute and then jumped to - this data dependency on the jump instructions can be entirely determined from a static analysis of the executable, no violation of fundamental computing science principles required.
So yeah, no real back and forth to the host platform.
AOT (ahead of time) really does work on whole binaries. That is also why the first launch of an Intel app seems longer sometimes. Intel binary in, ARM binary out.
You're still right that that's not sufficient, for example anything that generates Intel code will definitely need JIT (just in time) translation. But presumably a lot of code will still hit the happy AOP path.
That being said, a JIT does not have to be super slow. The early vwmware products, back before there was virtualization support in Intel products, actually had to do some translation as well: https://www.vmware.com/pdf/asplos235_adams.pdf
> That is also why the first launch of an Intel app seems longer sometimes. Intel binary in, ARM binary out.
I mean, we can call it an ARM binary or we can call it an instruction cache. I generally prefer the latter term, because what Rosetta produces are not standalone executables, they're incomplete. I don't know how often the happy path is used, but Rosetta can always be observed doing work at runtime.
JITs are great and Rosetta 2 is incredible! I just can't imagine it working over any sort of shared filesystem, that would add an incredible amount of latency.
An interpreter is still producing x86 instructions at some point, right? Or else what does the CPU execute? Am I totally misunderstanding how interpreters work?
> An interpreter is still producing x86 instructions at some point, right?
Not dynamically. They just call predefined C (or whatever the interpreter was written in) functions based on some internal mechanism.
> Or else what does the CPU execute?
Usually either the interpreter is just walking the AST and calling C functions based on the parse tree’s node type (this is very slow), or it will convert the AST into an opcode stream (not x86-64 opcodes, just internal names for integers, like OP_ADD = 0, OP_SUB = 1, etc) when parsing the file, and then the interpreter’s “core” will look something like a gigantic switch state statement with case OP_ADD: add(lhs, rhs) type cases. “add” in this case being a C function that implements the add semantics in this language. (The latter approach, where the input file is converted to some intermediate form for more efficient execution after the parse tree is derived, is more properly termed a virtual machine and “interpreter” generally only refers to the AST approach. People tend to use “interpreter” pretty broadly in informal conversations, but Python is strictly speaking a VM, not an interpreter)
In either case, the only thing emitting x86-64 is the compiler that built the interpreter’s binary.
> Am I totally misunderstanding how interpreters work?
You’re confusing them with JITs.
If every interpreter had to roll their own dynamic binary generation, they’d be a hell of a lot less portable (like JITs).
Rosetta is very impressive! I just don't see how they could maintain that by passing instructions back and fourth over a shared drive, that would be a ridiculous amount of latency!
The shared drive is just a licensing trick. They do an ioctl over /proc/self/exe as the licensing mechanism. (and that's routed over with virtio-fs to the host)
They are exporting some sort of Linux ARM binary under a virtual filesystem mount point that handles execution of x64 images.
Probably, that binary is passing the instructions into native macOS Rosetta code for translation, but its also possible that the entire Rosetta code was ported to Linux.
Sort of, yes-- there is `binfmt_misc` handling of PE excutables and virtual filesystems (akin to VirtIOFS) involved, but no binary/architecture translation like Rosetta.
Does anyone know some work in the opposite direction, that is, running OSX applications with x86 CPUs? I'm seeing a lot of effort with making the Apple's hardware to be universal and run everything else (like Asahi Linux and Rosetta), while I can only imagine how impossible it is becoming to emulate the M1/ARM and run a hackintosh.
Semi-related question for those who might know: Is it possible to run, say, Debian for x86 in a VM on M1 Macs? If so, I assume the performance is low because it has to translate every x86 instruction to ARM?
I know of Azure SQL Edge, which has significant limitations, but I don't know how to run MS SQL Server on an M1 at anything more than glacial speeds (i.e. full system emulation in Qemu). Is there something I don't know about? Please share!
I think it’s already the case. The kernel is arm64 and the amd64 containers are running with qemu. Most containers are fine but I experienced some issues, with maintainers refusing to support this setup.
I understand that you can replace qemu by rosetta now.
Containers are virtualized on macOS. x86-64 containers have used qemu to run the whole system under emulation, which is pretty slow. This workflow continues to require QEMU; however, now you can create an arm64 container and run x86-64 code in that using this technology.
What is arm64 container? Container is just a namespaced process so if your binary is x86 it is still technically x86 container just run using emulation. And qemu user static already does translation a la rosetta 2 so there probably wont be much difference in speed
If by plain qemu you mean its user mode emulation then i think it remains to be seen what the performance improvement will be. User mode already can be ok if it’s not doing endianness translation
I'm gonna guess that if Apple invested this level of effort into it, it's not a "remains to be seen what the performance improvement will be". You can take their graphs and charts and sort of de-reality-distortion-field them to know that it will be much faster than qemu for the workloads Apple knows people run frequently, but still slower than if you had a new x86_64 machine.
I've done a lot of work with running userland qemu for x86_64 in aarch64 vms on macOS 12. The experience is serviceable for simple things, but if I build anything really heavyweight, I find I need a native arch builder. I have little doubt that Rosetta will be an improvement.
Correct, on macs it spawns a (usually aarch64) vm using apples hypervisor framework api (hw virt) and runs docker daemon inside of it. Then it’s what I described above - just an process running arm instructions. If you’re using qemu binfmt_misc runner which uses “user mode emulation” not to be confused with system emulation which does full blown hw emulation. So it’s aready doing similar things to rosetta 2
Yes qemu and Rosetta perform the same basic role here (bintfmt handler for x86 bins), but Rosetta should perform significantly better (otherwise there would be virtually no reason at all to build this).
Something needs to set up the VM that Docker runs in. Docker Desktop is that something. The GUI even lets you tweak the VM resources. I suppose you could do it all on your own, but most macOS users probably just install Docker Desktop and let it handle the VM.
Op of thread here: there seems to be a lot of back and forth on running Arm64 containers and executing Rosetta in them to run something x86 but as far as I can tell (correct me if I’m mistaken and I think I might be) that doesn’t really help because my production containers are x86 so I would need to compile my container for Arm64 then run Rosetta in which is a totally divergent image and kind of useless for running production images locally.
The container’s kernel has to run on hardware (that’s what kernels do). If you have an x86 container that means an x86 kernel. Rosetta only runs user mode processes (and presumably in this environment knows how to translate linux system calls?).
If you /have/ to run an x86 kernel (I’m not sure what the reason would be?) you’re stuck with full system emulation a la qemu.
In reality what you probably have is a few x86 binaries that you need, and it sounds like this will let docker, VMware, qemu, whatever, leverage Rosetta w/o having to implement it themselves.
Now, if you’re running something that does codegen at runtime, then Rosetta is pretty much just as screwed as any other tech you might want to use.
Can Apple please just make a simple Virtualization front end ala libvirts Virtual Machine Manager. They seem to have all the bits, just need to bring them all together.
Seems like they're leaving that part up to the dev community (which I'm fine with). They'll never be able to make every person and every use case happy, so better to just give us the tools to do so ourselves. Check out UTM if you want a GUI for hypervisor.framework powered VMs.
I investigated virtualization framework recently and I'm kind of impressed. They really implemented a lot of things. And coming updates makes it even better.
Here's what I've learned:
1. You can run console Linux. But you need to download ARM ISO, extract kernel and initrd and find out exact kernel parameters. Not a big issue, but probably can't be done automatically. You can't just boot ISO.
2. Disk support is bad. No snapshots, etc. Just single file. I think that you can emulate snapshots with APFS CoW support, but that's not as good as something like qcow2.
3. No GUI support (for Linux, for macOS there's GUI support but I have no idea how it works).
4. Network support is limited. You can run NAT but then you can't communicate with host. You can run bridge, but I don't like this mode and didn't even try it. You can implement userspace networking completely by yourself, but that's a lot of work and I didn't try it either. But I think that if someone would want to build a virtualization frontend, he had to do it. Otherwise basic features like port mapping will not be available.
Now from what I read at this topic: ISO boot is coming, UEFI support is coming, Linux GUI support is coming. And I guess with enough tinkering it should be possible to run other BSDs, as they just leverage standard virtio protocols.
Also Windows virtualization theoretically should be possible. I mean it's just another UEFI OS. You would need to put virtio drivers, but Redhat wrote virtio drivers (for x86, but should be possible to compile for ARM).
I just wish that this virtualization framework was more extensible. Right now it's like use it or drop it. For example I didn't find any way to implement qcow2 support while using it for other things.
how feasible is the x86 windows virtualization? ive looked into UTM but it seems to be quite slow, how easy would it be to port this to windows then? "theoretically possible" could mean anything, but its a shot in the dark, haha.
I think this WWDC session will discuss that exact scenario. From the short description, I think they are still going to let the community handle the wiring of this into a VM management application. But, hopefully, this tutorial will make this process much clearer. I’ve tried to follow some of the older Framework examples, but I didn’t make it very far.
Try to simplify my understanding- can we position of apple on a) doing wsl1 or b) even less just provide tool to do wsl1 or c) … wsl2 … tool only to wsl2 …
Or we have a simpler mental model ? From a non-system land programmer point of view ?
Somehow, I wish there was an easy way to do the opposite: creating and running arm images in intel laptops.
This would allow me to run images easily and safely in Graviton based ec2 instances.
Isn't Rosetta 2 the opposite of libhoundini? (translating from x86 to ARM, instead of ARM to x86), I also fail to see how an application that has its roots in a PPC to x86 translator (Rosetta 1) from 2006, is somehow a rip off of an application from around 5 years ago.
See DEC's FX!32 (running x86 WinNT 3.51 binaries on WinNT 3.51 workstations with Alpha AXP CPUs) and HP's Project Dynamo. There are probably even earlier examples, but FX!32 was roughly equivalent to Rosetta, and shipping production code in the 1990s.
interesting. so this fixes the major performance issue by allowing users to install a user installable rosetta inside user created vms.
could still end up with some "works for me" but still broken on prod issues resulting from the underlying architecture being different (for intel backing infra), and also some questions around how it would work in practice in terms of integrating with production container workflows, but seems like a boon for anyone who is struggling today with intel vm or container performance issues on apple silicon.
Basic question: Why is this faster than running Intel Linux apps in an emulated Intel Linux VM? Because Rosetta is faster than QEMU, and you only need to emulate one application rather than the entire kernel?
Emulation of an x86 kernel level means you lose the hardware-assisted virtualization support you'd get with an ARM kernel, and emulating an MMU is slow (among other things.)
Technically this would be replacing QEMU user-mode emulation. Which isn't fast in a large part because QEMU being portable to all host architectures was more important than speed.
a lot of the performance gains in rosetta 2 come from load time translation of executables and libraries. so when you run a binary on the host mac, rosetta jumps in, does a one time translation to something that can run natively on the mx processor and then (probably) caches the result on the filesystem. next time you run it, it runs the cached translation. if you're running a vm without support inside the guest for this, then you're just running a big intel blob on the mx processor and having to do realtime translation which is really slow. (worst case, you have an interrupt for every instruction, as you have to translate it... although i assume it must be better than that. either way you're constantly interrupting and context switching between the target code you're trying to run and the translation environment, context switches are expensive in and of themselves, and they also defeat a lot of caching)
There is a specific piece of x86-64 software I need for my job, a back end service. It meant there was no way for me to use an M1 Mac for my job. I wish this was available last year, maybe I could’ve upgraded to an M1 based Mac.
when the m1 initially shipped, i was watching this issue like a hawk. rosetta was really cool technology, but they seemed to really miss the mark in terms of many developer workflows that involve intel vms and containers.
it was particularly interesting to me as i always got the sense that a lot of the success of the intel era macs was carried by their popularity and (and recommendability) amongst the internet development and engineering crowd. it seemed to me like a mistake to potentially alienate that group.
I figured it was largely a time management thing. They could only do so much (I mean they moved the whole base OS architecture… again).
Glad to see it.
Hope that by the time I get my next work laptop it’s still there, or better yet we’ve moved to a newer version of the software where this isn’t a problem.
Ime podman (or docker) + qemu-user-static covers nearly everything. There are some issues with older go binaries emulation but most of c/c++ x86 code works as is
I think it depends greatly on what instructions are being used (sse, avx, etc vs neon). Some benches I’ve seen boast rosetta2 is only 30% hit but that’s doubtful to me based on my own experience running factorio =)
I don't know about "production container workflows" on a Mac, however the only thing that needs to be done is to use the new Rosetta binfmt handlers instead of qemu (which Docker Desktop already sets up for you).
I imagine Docker will have an option to use Rosetta instead of qemu.
> The remaining steps required to activate Rosetta in a Linux guest aren’t commands that your app can execute or that you can script from inside your application to a Linux VM; the user must perform them either interactively or as part of a script while logged in to the Linux guest. You must communicate these requirements to the user of your app.
Most likely Docker Desktop would have an option to toggle between Rosetta and qemu.
Docker Desktop manages the vm, so it would be up to it to mount the Rosetta share into the vm and register it with binfmt.
There will be a session posted tomorrow from Apple on how to use their sample to run a Linux VM with Rosetta support. I wonder if the sample on this page will do the trick by itself?
Virtualization.framework is the hypervisor itself, you only need simple tool to launch it. You probably could just copy-paste provided code into eg. https://github.com/gyf304/vmcli. However macOS 13 beta currently seems to be only available for registered developers.
It seems that very little on the part of the hypervisor is required, so it shouldn't take them long to add it. They basically just have to call a function to create the virtual filesystem device for the guest.
How would this help windows specifically, or are you thinking steam+wine? In which case presumably steam+wine on Mac would have lower perf overhead.
If your thinking involves a windows VM under linux you’re SOOL, as that has the same problem as an win/x86 vm on Mac: it’s a full system emulation - eg qemu, etc
Rosetta can translate user mode code, because every can either be translated to native code /or/ is a system call, and apparently Rosetta knows how to translate linux syscalls.
You’d surely only need that if you had significant raw assembly to translate, in which case you’d have to assume that there were technical reasons for exact memory layout in x86, or perf reasons where you’d presumably want handcoded arm64 assembly?
Yes, that is exactly the case. I am involved in the effort to port Clozure Common Lisp to the M1 and it has a lot of bespoke assembly code.
(The x86_64 version of CCL actually works under Rosetta (more or less) which absolutely blows my mind since it's a compiler that emits x86 code at run-time!)
One issue you'd hit is that Rosetta is going to do fairly linear translation, and isn't going to improve calling conventions to the native one (in practice not terrible as x86_64 mercifully has enough registers for the all arguments in the general case).
In a sense if you were to go through the hand assembly one at a time and do 1:1 substitution with ARMv8 equivalent you'd probably be able to do a bit better, just because you don't have to consider behaviour of generic x86 code.
I believe the main issue is file syncing between macOS and the container host OS.
I don’t have a M1 mac, but on my Mac a standard Linux VM has no impact on performance or battery life. The minute I use Docker the machine become sluggish and halves (literally) the batter life, I now avoids is as much as possible.
I have a M1 Macbook Pro with 16 GB RAM, and qemu ballons in RAM when I try to stand up our backend microservice stack locally - it uses RAM until there's none left, and falls over. My teammates running Linux report ~30mb of RAM usage by the Docker processes to run the same containers.
I don't think that's right - containers are just userspace code, nothing to boot.
By my reading of TFA, your container runtime (eg Docker for Mac) should be able to provide an ARM VM, and use this Rosetta feature to run amd64 containers directly.
Containers that you and everyone else think of use Linux distros as a base, so yeah, anything else than Linux actually had to boot a VM to use them. That’s what eg Docker does on macOS.
Why does the rosetta binfmt_misc setup exclude starting amd64 containers? It's the same functionality the Docker desktop arm64 VM currently uses to run the process for an amd64 container via qemu (or any non arm64 container).
It's QEMU emulation. It's slow. It also segfaults a lot. I am able to run a select few containers on it, but I needed to sort out native arm64 for the vast majority.
The main reason for segfaults is that not all syscalls (syscalls on linux are different per architecture - not even 32bit and 64bit x86 use the same) are implemented in a 100% compatible way.
Yes, Docker on macOS can switch from qemu-user to this. It uses the same binfmt mechanism, so it should be mostly a drop-in replacement.
This is only available in combination with Virtualization.framework, but Docker is already migrating from Hypervisor.framework to Virtualization.framework.
I have to admit to a little bit of trepidation about the Rosetta mount point. Will this work for Docker without explicitly passing that mount point through to the containers?
The way this would work is Docker would be setting this up for you (maybe through some option to switch between qemu and Rosetta).
The containers should not need the mount.
What happens is Rosetta gets registered with the kernel (binfmt_misc) to execute x86 binaries.
The is the same mechanism that allows seamless qemu support.
Yeah I'm not entirely sure how they're going to make this work, but I kind of understand why it's so weird, because there's really not any very good way to communicate between macOS userspace and some arbitrary kernel running inside the VM…
Microsoft has a version of Windows (and even Visual Studio Pro) that runs on aarch64, specifically for Qualcomm SOCs, so it wouldn't be completely outside of possible. But I suspect they won't do it because they're probably be more interested in pushing Windows on ARM enthusiasts to Microsoft's and their partners' own hardware.
From what I've heard, Microsoft and Qualcomm signed an exclusive deal for Windows on ARM for a number of years. Microsoft could theoretically release a Windows version for the M1 but they'd be breaking their contract with Qualcomm which can only go down badly.
However, in a few years' time the deal runs out and it might be possible again, unless they can think of a reason to renew the deal (i.e. inclusion of Qualcomm's optimised x86/amd64 translation layer).
This is a huge overstatement. A few engineers went to Qualcomm, not the "Apple M1 team." Market forces at work. However, I do suspect they will help Qualcomm come closer to being competitive. How close they will get in performance remains to be seen. The M2 is already about 20% faster with the same number of cores.
> Nuvia was founded back in 2019 by former Apple silicon executive Gerard Williams III, along with Manu Gulati and John Bruno. Williams was the chief architect behind several major Apple CPUs and chipsets from 2010 to 2019 .. the Cyclone, Typhoon, Twister, Hurricane, Monsoon, Vortex, Lightning and Firestorm CPUs. These CPUs were featured in the Apple A7, A8, A9, A10, A11, A12 series, A13, and A14 respectively. The Nuvia founder’s profile also notes that he was the chief architect for Apple’s Mac hardware ... Going back even futher, the Nuvia co-founder worked at Arm from 1998 to 2010, working on CPU tech like the Arm Cortex-A8 and Cortex-A15 CPU cores.
> Qualcomm paid $1.4 billion for Nuvia, they must have hired more engineers and/or invented new IP.
This assumes that valuation is rational. It is not, and recent history is full of companies over-paying or under-paying for their acquisitions. The proof of the pudding will be in the eating, and the pudding will still take a while to get ready.
M1 is a fixed target, everyone will get there eventually. For example, AFAIK Intel has plans to have something competitive with the M1 in terms of performance per watt at some point in 2023. The question is whether Qualcomm will be able to get where Apple will be at that time, because nobody will care about “M1-equivalent” in 2 years.
And yeah, Nuvia is more “a handful of engineers” (though quite good at their job) than “the former Apple M1 team”. It took Apple billions upon billions and the better part of a decade. These things are very long term plans, and at this scale it depends at least as much on high-level strategies than on engineering.
Windows 11 for ARM apparently runs nicely in a VM on ARM Macs, a coworker of mine is using it for small projects. Including x86 emultion - his project requires some x86 tools.
[1] https://gist.github.com/akihikodaki/87df4149e7ca87f18dc56807...