I'm in game development, and I run ESXi for two reasons:
* Unmodified macOS guest VMs can run under ESXi (if you're reading this on macOS, you have an Apple-made VMXNet3 network adapter driver on your system--see /System/Library/Extensions/IONetworkingFamily.kext/Contents/PlugIns/AppleVmxnet3Ethernet.kext )
* Accelerated 3D has reasonable guest support, even as pure software. You wouldn't want to work in one of those guest VMs, but for any sort of build agent it should be fine, including opening i.e. Unity editor itself in-VM to diagnose something
Does anyone know where either of these things stand with Proxmox today?
I imagine macOS VM under Proxmox is basically a hackintosh with i.e. OpenCore as bootloader?
I run macOS/OSX Sonoma in Proxmox. It does pcie/gpu passthrough (AMD rx580 8gb). The proxmox host is running a AMD Ryzen 7 5700G cpu and has 64gb memory. The mac vm disks are on a zfs fs on a wd black nvme ssd.
It's the fastest mac I've ever owned and it's virtual, executed on a machine running a chip that apple never supported, and you'd never be able to tell it was a vm unless you were told so. kvm and vfio are amazing pieces of software.
Why? IIRC running ZFS on NVMe SSDs limit their performance seriously. With sufficient queue depth modern SSDs can easily get up to 1mln+ IOPS and on ZFS I can barely get 100k :(
Most probably because ZFS still has the most extensive data correctness guarantess of all filesystems on Linux. Yes, bcachefs will have checksums too but it it is still in beta.
copy on write, instant snapshots, data checksumming, easy rollbacks, auto trim, compression
proxmox takes advantage of zfs' features if available, like when you select block storage on zfs it makes each vm disk its own dataset which you can tune as you see fit for each one. cloning a vm is instant if you make a cow clone. suspending the vm to do snapshot based backups is also nearly instant, etc
Disable atimes and turn off the arc cache and metadata cache (don't really need either with fast storage compared to spinning rust) and use lz4 compression and you minimize write amplification or needless wear on the drive. zfs is perfectly fine on ssd for my use case
The lack of 3D paravirtual devices is a real sore spot in kvm. To my knowledge, virgl still isn't quite there but is all there is so far in this space. VMware has the best implementation IMO and everything else is a step down.
Running macOS is only supported/licensing-compliant on Apple-branded hardware anyway, and with the supported Intel Macs getting pretty old this was inevitable anyway.
If you’re just running a bare metal hypervisor then you may as well just go for a second hand tiny form factor platform. No point paying 15-20% extra for the same specs in a svelte case if you’re not running native macOS.
> No point paying 15-20% extra for the same specs in a svelte case if you’re not running native macOS.
The main point is proper non-hackintosh support for running (unlimited amount) of macOS vms on esxi - which requires Apple hardware. Running macOS, Windows and Linux on the metal is another benefit. IMO it's almost as versatile as a framework chromebook (chromeos, crostini, kvm, android with play store)
The SPICE backend has decent OpenGL 3D support with software rendering, it's slow but it works for simple graphics. It's intended for 2D so the desktop's pretty fast IMO. That only works for Linux and Windows guests though, not Apple ones.
MacOS VMs do work in Proxmox with a Hackintosh setup but you pretty much have to passthrough a GPU to the VM if you're using the GUI. Otherwise you're stuck with the Apple VNC remote desktop and that's unbearably slow.
For paravirtualized hardware rendering you can use virtio-gpu. In addition to Linux, a Windows guest driver is available but it's highly experimental still and not very easy to get working.
What's the best solution for remote viewing with virtio-gpu? I vaguely recall that it had some incompatibility with spice.
I have a bunch of qemu/kvm virtual machines for various tasks. For two of them I decided that graphics performance needed to be at least OK and ended up buying and passing through old Quadro cards. It'd be lovely to not have to do that.
I haven't really bothered with macos GPU acceleration, It is possible but a little fiddly with an nvidia card. I mostly rely on screen sharing to display the remote vm and that's really good (rich text copy/paste, drag and drop files, etc)
As to general guest 3d, you can do this with GPU passthrough, and although it's technical the first time, each vm is then easy.
basically I add this line to each VM for my graphics card:
hostpci0: 01:00,pcie=1,x-vga=1
and this passthroughs a USB controller for my USB dac:
hostpci1: 03:00,pcie=1
(this is specific to my system device tree, it will usually be different for each specific machine)
The passthough has worked wonderfully for Windows, ubuntu and arch linux guests, giving a full accelerated desktop, and smooth gaming with non-choppy sound.
two things I wish proxmox did better:
- passthrough USB into containers
usb is pretty fiddly to propagate into a container. It is marginally better with VMs but sometimes devices still don't show up.
- docker/podmain containers
proxmox has lxc containers, but using them is a little like being a sysadmin. Docker/podmain containers that you can build from a dockerfile would be a much better experience.
Nested virtualization also works great on ESXi for macOS guests ( so you can run docker desktop if so inclined ). I believe this is possible with proxmox as well with CPU=host but have not tried it.
Apple has been adding a lot of virt and SPICE things IIRC. Some of it isn't supported in VMware (it lacks a bunch of standard virt support), but the facilities are growing instead of shrinking which is a good sign.
On Proxmox you can do the same. You're going to need OpenCore if you're not on a Mac indeed. But if you're not on a Mac you're breaking the EULA anyway.
Mostly used it when trying to track down reported macOS bugs for an OSS project, so maybe once every few months. But it's worked quite well at those times. :)
quickemu also does the job, not just for macOS but many operating systems
"Quickly create and run highly optimised desktop virtual machines for Linux, macOS and Windows; with just two commands. You decide what operating system you want to run and Quickemu will figure out the best way to do it for you."
https://github.com/quickemu-project/quickemu
* Unmodified macOS guest VMs can run under ESXi (if you're reading this on macOS, you have an Apple-made VMXNet3 network adapter driver on your system--see /System/Library/Extensions/IONetworkingFamily.kext/Contents/PlugIns/AppleVmxnet3Ethernet.kext )
* Accelerated 3D has reasonable guest support, even as pure software. You wouldn't want to work in one of those guest VMs, but for any sort of build agent it should be fine, including opening i.e. Unity editor itself in-VM to diagnose something
Does anyone know where either of these things stand with Proxmox today?
I imagine macOS VM under Proxmox is basically a hackintosh with i.e. OpenCore as bootloader?