QEMU 6.2 introduces full hypervisor support for M1 Macs. I've installed ARM Ubuntu Server in a VM on my MacBook, and wrote a blog post explaining it. It uses libvirt to start the VM in headless mode.
> It uses libvirt to start the VM in headless mode.
So then, I could be running virt-manager and manage VMs on an M1? That would slide nicely into my existing virt-manager usage. But, not with a soldered-in SSD. I tend to keep machines running a LONG time, and I'd hate to pitch an otherwise perfectly-good machine because the SSD wore out and couldn't be replaced. The "expensive disposable" situation with cell phones is bad enough, let's not have this in our PCs as well.
As a counterpoint, the only SSDs I've ever had fail on me were some early OCZ units (big surprise), but I still check the SMART data on drives I'm using and see the wear indicators slowly advancing. I keep stuff backed up on ZFS spinning rust.
I'm sure Apple will introduce a good "SSD Replacement" program when they wear out. We'll be without our machines for a few weeks, it'll cost $200 more than it should - but they will find a way to replace them and stop everyone complaining.
Thanks for that blog post! I downloaded https://mac.getutm.app/ a couple of weeks ago but haven't used it yet - this sounds similar but a more automated way to do what you do?
Coming from Windows, Vagrant & Virtualbox to a M1 Mac I'm looking to find an easy way to manage multiple VMs for developing different projects.
I'm using utm to run a ubuntu server that where I do all my dev work. Since it is a vm I am a lot less scared of messing up my main system. It is very convenient with vscode remote, which only needed to be configured once and will automatically connect to the vm.
And for networking, use the virtio drivers for Windows, after installing those, you can remove usb and switch to -device virtio-keyboard -device virtio-tablet for input too.
PowerPC versions of Mac OS X run well under QEMU/UTM and Apple silicon. Cobbled together this guide using Tiger as the guest: https://tinyapps.org/docs/tiger-on-m1.html . More than performant enough for running OED[1] and MPEG Streamclip[2], two old favorites.
I'd love to understand virtualization/emulation better, on the technical level. If I understand correctly, in modern CPUs it's all done with the help of specialized support in the instruction set, but I've never learned in depth
- how that works
- how it differs between Intel/AMD/ARM
- whether there are competing approaches
- what are the main software project "players" in the field
- which of them are open-source with humanly understandable
source code
Is there a summary/tutorial about all this stuff someone could recommend (or write)?
Well, yes, you can do that, but Intel's Software Developer's Manuals are terribly written. They are good as references for details, but only after you understand the grand picture. Learning something new from them is quite a torturous process.
Emulations is pretty much literally just mapping instructions between processors. So there may be an instruction in my custom chipset called "Add4", which adds 4 inputs. I would emulate ADD4 RAX, 0x1234, 0x2345, 0x3456 that by
It gets a bit more complicated with architecture differences like memory configurations. But that all emulation is.
When you're virtualizing, you pretty much just need to manage hardware. The hypervisor does this for you by managing which resources go to where. You could virtualize it by just running it like a program. But that's really painful and tedious, so you rely on the CPU to support it. Each chip has it's differences, but it's effectively just like a syscall. You have VMCALL and VMEXIT instructions. And then you have a handler in your vmexit table, which is exactly like a syscall table. So if(exitreason == CPUID_EXIT) cpuid_handler();
I know next to nothing about hardware / instruction set support from the CPU side, but there are several hypervisors who are open source and thus available for study: Qemu itself, Xen, Linux KVM, FreeBSD's bhyve, OpenBSD's vmm. Oh, and there's VirtualBox.
I vaguely recall someone telling me that the qemu source code is fairly readable, but I have not looked at any of these myself, so consider it hearsay.
> I vaguely recall someone telling me that the qemu source code is fairly readable [...]
That someone was either pranking you or knows a mysterious part of the qemu codebase that's unlike the rest of the criminally underdocumented qemu soup. Or my standards are too high.
The source is okay, the problem is that there's really no docs/examples for even the basic stuff. Do you want to boot from a virtual block device and PXE from a network drive ... well, maybe these 300 command line arguments in some random order will work. Eventually. Have fun. So reading the source is a big help in that. But it's not ideal. And libvirt/oVirt/etc are helpful, but not for just that quick and dirty "let's try something out with the linux kernel in qemu" thing.
a quick bird's eye view summary to get your bearings:
(caveat, I'm taking shortcuts so things may not survive scrutiny but it should be good enough for an overall understanding and getting down to the truth if you so desire)
emulation: software that implements and simulate hardware to mimic it as expected by software that would run within emulation
this is slow, because you have to really pretend hardware down to the details (simulate clocks, irqs, video output, etc). often you can get away with not being 100% exact and accurate depending on the emulated software's assumptions or you don't care if some specific output is incorrect (e.g graphical difference or oddity in games), which gives you some nice speedups. 100% accurate emulators can and do exist but you can see the difference in requirements (see mesen, higan) with inaccurate emulators.
some people quickly figured out that if you are running software compiled for a specific target (e.g a x86_64 CPU) that is the same target as the host then it kinda doesn't make sense to emulate the CPU... and that you can kinda obviously pass through cpu instructions instead of simulating the very hardware you're running on... this is:
virtualization: instead of emulating, expose a non-emulated device from the host that the software is known or expected to be compatible with, thus saving a lot of cycles.
for the CPUs this is typically done through an instruction that creates an isolated context in which the software will run. reminder: a CPU has multiple "rings" already under which software is run, for security and stability reasons (memory protection, etc...), e.g the kernel runs under ring 0 and userland processes are under ring 1, each in a different context preventing one to access the other in the same ring, and to go up rings, all checked at the hardware level. the virtualization instructions roughly do the same thing and nest things so that a virtual cpu runs code in a separate ring+context, in a way that the guest OS thinks it's alone. see also type 1 (xen, hyper-v) vs type 2 (qemu, vmware, parallels) hypervisors.
other devices can be implemented in lightweight, efficient ways, and exposed as virtual devices to the virtual machine for the nested OS to pick up. e.g instead of emulating a full PCI network card complete with pseudophysical hardware matching a real one, one can have a simpler device exposed that merely takes ethernet packets and inject them into the host kernel's network stack. or same for IO, instead of emulating a sata device complete with commands and all, one could just expose a block device and ask the host kernel to read its data from some host file directly. this requires dedicated drivers for these virtual devices in the guest OS though.
virtualization is as good as native because often it's literally native, only with hardware sandboxes and maybe a few added indirect wrappers, that Windows on PCs and apps and games on Xboxes actually run in VMs under HyperV all the time!
so back to the CPU side of things, you can notably see how it is impossible with virtualisation for a CPU to execute anything else than its own instruction set.
now, executing foreign instructions more efficiently than literally interpreting them can be achieved through various means such as binary translation or dynamic recompilation, but it still has to take some guest assumptions into account, e.g Rosetta 2 (which is neither virtualisation nor emulation) translates x64 code to arm64 but x64 assumes a certain ordering of operations between cpu and memory that just cannot work as is on arm64, so there's a special Apple-specific instruction to switch a CPU thread to a different memory model, still running arm64 instructions (translated from x64) but in a way that matches the x64 behaviour.
there are many more gritty details, and things are not always as clear cut as I described (or flat out lied about, the Feynman way) but it should be a good enough model to start with.
Nice, I've been using qemu on my mac as am alternative to docker. I run arch linux in a vm with docker installed and an ssh daemon running on port 2222. On my mac, I simply point docker cli to use the vm by setting DOCKER_HOST=ssh://jilles@localhost:2222
Works great and it cut me loose from docker for mac. The networking was a bit of a pain in the ass to get going until I figured out that it basically does not work without libvirt installed as well. Basically none of the networking options do anything until you do that (I've tried them all).
> Basically none of the networking options do anything until you do that (I've tried them all).
Can you share more?
Is it completely impossible to make networking function in vm let's say debian x86 vm running on mac m1 without libvirt or it's very hard to configure?
Don't tend to use those for development. But I imagine they would end up mounting directories inside the vm. You could get to those via scp or something like that.
The problem tends to be getting inotify to work properly across the boundary. You want to propagate filesystem changes into the VM, in the ideal world. Docker-desktop's got some closed-source wizardry that makes it Just Work, I'm just not sure what's as slick in the open-source ecosystem.
Unless Apple implements some kind of support for the x86 ISA in its CPUs (which is unrealistic), hvf will always only support arm and aarch64 guests. Running a non-native architecture requires emulation, which will always be slower than just running a guest on the native CPU.
This, exactly. Rosetta 2 emulates x86_64 Darwin binaries on AArch64 Darwin - it means that tricks like jumping to native code as soon as possible are feasible.
For instance, you can call a native counterpart of a system library instead of emulating it down to the syscall level, and generate native code that calls into that.
Running kernel level code requires you to emulate an entire machine, which is way more complicated than just emulating the userland like Rosetta does. Running an unmodified OS kernel requires emulation of stuff like interrupts, firmware, peripherals ... Yes, you can dynamically JIT compile some codepaths into native code, which is something that AFAIK most emulator already do, but there's still a much higher performance cap than with something like Rosetta or Wine.
Great news, QEMU is really great. Will have to give this a spin on the MiSTer again. I was experimenting with emulating the 68K in qemu in combination with FPGA reimplementation of the Amiga chipset. I ran into performance issues with the way AmigaOS manages memory, where stack and code can be in the same page. Qemu seems very inefficient in that case since it constantly invalidates the translated code I think, I wonder if that aspect has been improved at all.
https://www.naut.ca/blog/2021/12/09/arm64-vm-on-macos-with-l...
And this is the script to force QEMU to run on the M1 efficiency cores to save battery life:
https://github.com/yoonsikp/vm_configs/blob/master/power_sav...