I'm really sold on the idea: Instead of a full-blown OS, you compile your application with a thin layer of support libraries that provide the OS features that your application needs (network, I/O) and that talks to a hypervisor.
I mean, if your application runs in a virtualized environment, there's little need to SSH into the system in the first place (except for debugging purposes). Thus, why bother with a full-blown operating system? In the virtualized case, the true OS logic is in the host OS anyway, talking to the hardware. Cutting out all those superfluous layers in the app VM makes it small, start quickly, and gives less attack surface. Sounds like a win-win to me.
In contrast, FreeBSD on Firecracker is a full-blown OS, but boots in 25 milliseconds on the Firecracker hypervisor.
To be a bit more precise, what we achieved so far: apart from web servers and TLS reverse proxies (with DNS-backed let's encrypt provisioning), DNS authoritative servers, a CalDAV server, git client and servers, SMTP stack (including dkim etc.), OpenVPN implementation, archive mirror (using tar as file system), monitoring with syslog and influx, ...
In those kind of corporate environments liability is a concern, and something like MirageOS would be quite appealing.
These are the same kind of environments where CI/CD can only fetch from internal repos, where stuff only gets installed after IT and legal had a say on it.
I haven’t SSH’d into a machine/container to debug or fix a production issue in years. You flat out can’t even SSH into anything on the last few prod setups I’ve worked on: permissions prevent it, and most containers have sshd and friends stripped out.
I would love a super slimmed down stack to run workloads on. Most of the time when I run workloads, I want to run the workload and nothing else. The OS in the container exists to start the binary I care about and then basically very little else-these applications aren’t making use of a scheduler, or the init system (because again, I’m only running my binary), users and permissions go unused a good chunk of the time. With stuff like quic becoming more popular we’ve got user space networking too, so that’s out. So by this stage we’ve got a whole OSCthats doing almost nothing except passing stuff between layers an incurring a performance and risk overhead.
If we could have the same functionality, with better performance and the same or better security, why wouldn’t I do that?
> This concept has been done multiple times and has always failed
Isn’t it likely that this idea was just “before its time”? The compute landscape these days is pretty different to ~20+ years ago: massively available public cloud compute, prolific container usage, etc. Software stacks are deeper, more ossified and less fully understood than ever, I definitely see solutions that reduce that complexity being appealing.
exactly this, debugging is a pretty big part when it comes to operational aspects.
but I feel like similar to kubernetes container injection systems. I suspect in future we would see dynamic injection into the unikernel where you can enable debugging environment instantly.
like the library unikernel exposes a plugin system that you can inject in future, this way the unikernel is lightweight and only gets bloated when you eventually want to debug something.
What you as well can do is record-replay -- well, at least there used to be this option, I'm now lost whether it has never been merged anywhere, or it is stuck in some PR somewhere. This was truly great - since the external interface is so thin, it is easily doable to dump all external events (API calls and returns) onto disk and replay one-by-one, inspecting the state.
A big difference in my opinion is that projects like Mirage does not try to reimplement the same hardware drivers as the host system and instead implements much simpler drivers for virtual hardware.
For example, how do you write to a block device? You make a hypercall with a "handle", offset into the block device, the number of blocks to write and a pointer to memory to be written. There's no pretending it's a spinny disk with sectors and heads or what have you.
That is just infinitely worse than calling write() on a file handle like a regular application.
The hypervisor + OS + application model is stupid. The hypervisor and OS are duplicative. If you can rewrite the application (as is needed for a unikernel design) there is no reason to use a hypervisor in your stack at all. You should just be running a OS directly on the bare metal with no hypervisor at all.
The only reason to prefer a unikernel is because you wrongly believe that hypervisors are a security boundary. Everything else about them is strictly inferior to a standard application on OS model.
One of the key insights for unikernels in general is that everything is virtualized now to begin with. The entire public cloud: AWS, GCP, Azure, etc. is virtualization with an api on top of it.
So unikernels take advantage of this fact to get the performance, security and ease of use benefits.
Hypervisors most definitely have better hardware-based isolation than a few processes running in linux. In fact the isolation is so good that the entire public clouds are built on this model.
This only works if your environment is bare metal to begin with. In some cases you may be in an environment where a lot of applications are proprietary, virtualized, or otherwise constrained in some other way. It is not an environment where you get to pick and choose how everything is setup. In such a case, having a distinct small application that fits as "an OS" in a virtualized environment can fit a lot better in, than trying to retrofit a baremetal containerized environment into a large enterprise datacenter where everything is already running Windows servers and proprietary services, managed by other people than yourself.
There is no meaningful security difference between commercial operating systems and hypervisors when considering technically competent, commercially-motivated attackers. No systems built on such insecure systems can stop even minor attacks staffed by a single-digit number of FTEs. Maybe there is a material difference, but it is irrelevant compared to their total inadequacy relative to the present threat landscape. The person who can climb Mount Everest is still no closer to achieving orbit than the person who can only climb a tree.
To move beyond the mere practical aspects, even theoretically you are wrong. The techniques needed to develop a secure hypervisor are basically exactly the same techniques needed to develop a secure operating system. They are almost trivially transferrable. If you can do one, you can do the other. So, again, no advantage to preferring a hypervisor based solution.
Even if I accepted the claim that the techniques are the same, it's irrelevant. It's simply a matter of fact that standard installs of commodity server operating systems have a significantly larger attack surface than a unikernel system. Sys loggers, email agents, inflexible and insecure user/group access control, and just general ambient authority. Unikernel systems have none of this cruft by default.
I think many designs in tech have two states between which they oscillate. We don't go back exactly to the same thing, we do improve on things, but things are very similar to the way they were done 20-30 years ago (or 40 in this case)
Well it doesn't make much sense with games and other user applications nowadays, but if you want efficiency and a smaller attack surface in a production environment, why not? People already use Docker containers that were stripped to the bare minimum for the same reasons.
Well, that is kind of what the maker community is doing with Arduino and ESP32, and what makes them much more interesting to hack around, as yet another UNIX clone.
> I'm really sold on the idea: Instead of a full-blown OS, you compile your application with a thin layer of support libraries that provide the OS features that your application needs
I'm really sad that unikernels never took off, even there seemed to be a lot of excitement around them a few years back.
The second wave is coming :) . At www.unikraft.org we're hard at work providing a Linux-API compatible unikernel (run unmodified apps/langs) and combining it with tooling like docker and k8s.
Absolutely, MirageOS is indeed pivotal in the development of unikernels. MirageOS not only developed the unikernel concept but also coined the term 'unikernel'.
MirageOS represents a significant shift in cloud and network computing, focusing on building highly specialized, secure, and efficient unikernels that streamline application deployment by shedding unnecessary components. It also puts extra focus on security, with most of the stack — including TCP and TLS — being rewritten in a type-safe language (OCaml).
Currently, MirageOS is expanding to include bare-metal and embedded systems, aiming to bring the same level of security and efficiency to this domain. The move towards embedded systems is a natural progression for MirageOS, given our emphasis on minimalism and security in environments where resources are limited and reliability is crucial. See for instance what is happening with SpaceOS: https://tarides.com/blog/2023-07-31-ocaml-in-space-welcome-s...
The catch here is Ocaml. You can design your unikernel "bare metal" in any way you want and make it do nothing but the essentials you need ... and design it in any language you want as long as that language is OCaml.
I have especially had hopes for the UniK [1] project, as it was/is written in Go AFAIK. I see now it incorporates work from the Mirage project as well. Not sure what is the status of this project anymore though.
Unik was just a build tool that utilized other projects like Rump, Mirage, IncludeOS, etc. It's now dead since Solo pivoted a very long time ago to service mesh/api gateways.
The GoRump port they use was from us and then we realized we needed to code our own from the ground up for many reasons so we wrote https://nanos.org (runs as a go unikernel in GCP).
I didn't read your message, so I just clicked and I found that great nice project from Solo.io (the company behind Gloo Edge, etc.). Freaking awesome... I thought:
> Not sure what is the status of this project anymore
I tried to search if there are easy ways to do bindings to other languages. There seems to be bindings to C which should make it easier to do bindings to other languages too, but I am not sure whether someone has done that work just yet. Until then, I really doubt this thing will fly. Great project, nothing to say, but OCaml, ...
> MirageOS is a library operating system that constructs unikernels for secure, high-performance network applications across a variety of cloud computing and mobile platforms.
At least for cloud deployments you don't need a full blown OS when all you want to do is run apps like NGINX or Redis. If you use a unikernel instead, then you severely (orders of magnitude) reduce cold boot times, memory consumption, server density (thousands on hw-isolated instances on a single server) and TCB.
This talk is great in its revealing of history, does it in a very meaningful way. I had few romances with OCaml over the last two decades but never had a broad view on the different phases of its evolution. Anil provides a lot of insights about that here
What is the benefit over using containers, as in Docker? Whether you use a container runtime or an actual hypervisor comes down to pretty much the same thing, operationally. Both keep your self-contained services alive and distributed. From the application perspective, a container also contains only those parts of an OS the app actually needs, and defers everything else to the host. The only caveat about MirageOS seems to be that your applications need to be written in OCaml, which is a neat language and all, but certainly not mainstream…
At least for cloud deployments you'll (for almost all cases) already have a hypervisor underneath to provide strong isolation. With that in place, ideally you'd run your application (ultimately the only thing you care about) as close to that hypervisor as possible. Instead, we have hypervisor, and then inside the VM the (say Linux) kernel, user-space, the container runtime, and finally the application.
With a unikernel the stack becomes hypervisor and a VM that has a very thin layer and then the application -- as close to the application running on the hypervisor as possible. This results in lots of gains in terms of minimal cold boot times, memory usage, server density (thousands on a single server), etc.
In fact, you don't need to see containers and unikernels as an either or choice: in fact, at Unikraft (another unikernel project) for development and local deployment we have support for Docker/Dockerfiles -- and then for deployment we provide a lean unikernel as described above.
It makes no sense for a microservice that does one simple thing to run on top of 10 million lines of 90s C code. Especially since a lot of that code has to do with hardware quirks that don’t exist in a hypervised environment.
A hypervised environment also has 10 million lines of 90s C code with a lot of code dealing with hardware quirks, it is called the hypervisor.
You are right that it makes no sense to have a hypervisor, OS, and application. The hypervsior and OS are basically doing the same job. But the solution is not bare metal + hypervisor + device drivers + hypervisor services + library OS + application like a unikernel design. It is bare metal + OS + device drivers + OS services + application like a container design.
Some MirageOS developers were hired/acquired by docker -- though not the (open source) project, beither the code... There's still quite some work on MirageOS itself, including reproducible binary builds, VPN, DNS services, orchestration solutions, ...
Lots of them. VPS typically are not secured at the os level, but as separate VMs. For the cloud provider it's just a matter of picking an ISO (plus a few management extensions but those are usually optional)
No, CloudCaptain is based on Linux, and tries to provide a minimal, though Linux-based, image.
Unikernels do not use Linux at all: you can always try to minimize Linux but fundamentally it is a monolithic OS and fully specializing with it would require non-negligible engineering.
Instead, unikernels are (typically) based on a modular OS that makes it easier to pick and choose modules for each target application, resulting in images that can be an order of magnitude smaller, boot much faster, etc.
The difficulty with unikernels in the past has been to (1) making them Linux API compatible, (2) making them accessible/easy to use and (3) integrating them with popular tooling ecosystems (e.g., Docker, Kubernetes, Prometheus, etc.)
It is possible to run Mirage in ARM under for example KVM or using the seccomp target.
There is as well an experimental bare-metal target for raspberry pi 4 called gilbraltar https://github.com/dinosaure/gilbraltar. A big obstacle there is the device drivers. It is very cool to run bare metal on an rpi4, but it would be cool to be able use the network interface too.
I not always agree with Rob Pike, but this one is a must.
"We really are using a 1970s era operating system well past its sell-by date. We get a lot done, and we have fun, but let's face it, the fundamental design of Unix is older than many of the readers of Slashdot, while lots of different, great ideas about computing and networks have been developed in the last 30 years. Using Unix is the computing equivalent of listening only to music by David Cassidy."
We could’ve gotten something more like a simpler Swift if some attention was paid to state of the art in language research and implementation. No nil, composable errors, integrated generics, sum types, etc.
If you like language archeology, compare K&R C with the state of systems programming languages outside Bell Labs, in regards to safety and language features, since JOVIAL was introduced in 1958.
Hence why there is a certain irony towards the opinion UNIX is done, yet the efforts towards language design are as they are.
Plan 9, Inferno and Limbo came to be, and that was it.
If I understood it correctly, is more like something to run inside firecracker.
Is like a toolkit to build a really small os to run a single application, that then you run on top of a hypervisor.
No, though you could use Firecracker to launch a Mirage (or any other) unikernel.
Basically from the bottom up the stack is:
1. Hypervisor (e.g., KVM, Xen, Hyper-V), runs directly on the hardware
2. Virtual Machine Monitor (e.g., QEMU, Firecracker), running on the host's user-space (say Linux) and in charge of starting/stopping/managing VMs and interacting with the hypervisor
3. Virtual machines, eg, a Linux VM running an NGINX web server.
(the above is simplified because there are differences between type-1 and type-2 hypervisors, but those diffs would make this message too long)
A unikernel is actually a virtual machine, just a very specialized one that doesn't use a general-purpose OS underneath. They tend to use library OSes, so that it's possible to choose libs that are appropriate to each app at build time.
And while we're at it :) , a MicroVM is nothing more than a standard VM (e.g., based on Linux) launched/managed via a fast/modern VMM like Firecracker.
I mean, if your application runs in a virtualized environment, there's little need to SSH into the system in the first place (except for debugging purposes). Thus, why bother with a full-blown operating system? In the virtualized case, the true OS logic is in the host OS anyway, talking to the hardware. Cutting out all those superfluous layers in the app VM makes it small, start quickly, and gives less attack surface. Sounds like a win-win to me.
In contrast, FreeBSD on Firecracker is a full-blown OS, but boots in 25 milliseconds on the Firecracker hypervisor.