The issue with unikernels and things like Firecracker are that you can't run them on already-virtualized platforms
I researched Firecracker when I was looking for an alternative to Docker for deploying FaaS functions on an OpenFaaS-like clone I was building
It would have worked great if the target deployment was bare metal but if you're asking a user to deploy on IE an EC2 or Fargate or whatnot, you can't use these things so all points are moot
This is relevant if you're self-hosting or you ARE a service provider I guess.
(Yes, I know about Firecracker-in-Docker, but I mean real production use)
This is a limitation of whatever virtualized instance you're running on, not Firecracker itself. Firecracker depends on KVM, and AWS EC2 virtualized instances don't enable KVM. But not all virtualized instance services disable KVM.
Obviously, Firecracker being developed by AWS and AWS disabling KVM is not ideal :)
Google Cloud, for instance, allows nested virtualization, IIRC.
Ive used GCP nested virtualization. You pay for that overhead in performance so I wouldn't recommend it without more investigation. We were trying to simulate using luks and the physical is key insert / removal. Would have used it more if we could get GPU passthrough working
Yeah but imagine trying to convince people to use an OSS tool where the catch is that you have to deploy it on special instances, only on providers that support nested virtualization
Not a great DX, haha
I wound up using GraalVM's "Polyglot" abilities alongside it's WASM stuff
eh, I think most of the reason we don't see NV support on most public cloud instances is that nobody has come up with a compelling OSS tool that needs it. afaik most providers disable it because they want to discourage you from subletting their instances lol.
> The issue with unikernels and things like Firecracker are that you can't run them on already-virtualized platforms
I'm not sure about Firecrackers and it has to be enabled on the platform level but there are tricks you can use to run pseudo-nested VMs that look like they are nested without actually incurring into any nested virtualization overhead. This is something that some of us (crosvm) are exploring and playing around with a virtio-vhost-user (VVU) [0] proxy to basically set up a "sibling" (not nested) VM that runs its virtio device drivers into another VM without incurring into nested virtualization issues.
Firecrackers being originally based on a fork of crosvm might even benefit from this if they ever decide to play around this space, but also it'd need to be enabled on the platform level.
It's funny seeing stuff like a VVU implementation of vhost-vsock effectively run on a second VM and provide what seems to be a nested virtualization environment between two completely separate VMs. You get funky stuff like the other sibling VM's host address (cid = 2) becomes a non-parent VM so it looks 100% like a nested VM without nested virtualization performance hits.
This is a very common misunderstanding in how these actually get deployed in real life.
Disclosure: I work with the OPS/Nanos toolchain so work with people that deploy unikernels in production.
When we deploy them to AWS/GCP/Azure/etc. we are not managing the networking/storage/etc. like a k8s would do - we push all that responsibility back onto the cloud layer itself. So when you spin up a Nanos instance it spins up as its own EC2 instance with only your application - no linux, no k8s, nothing. The networking used is the networking provided by the vpc. You can configure it all you want but you aren't managing it. Now if you have your own infrastructure - knock yourselves out but for those already in the public clouds this is the preferred route. We essentially treat the vm as the application and the cloud as the operating system.
This allows you to have a lot better performance/security and it removes a ton of devops/sysadmin work.
I researched Firecracker when I was looking for an alternative to Docker for deploying FaaS functions on an OpenFaaS-like clone I was building
It would have worked great if the target deployment was bare metal but if you're asking a user to deploy on IE an EC2 or Fargate or whatnot, you can't use these things so all points are moot
This is relevant if you're self-hosting or you ARE a service provider I guess.
(Yes, I know about Firecracker-in-Docker, but I mean real production use)