I think it depends on the use case. The power of docker is that you don't need virtual machines with set memory reservations, so configuring applications to grow and shrink memory from a shared pool is a lot easier. Not requiring an extra kernel is also a positive because every kernel needs to do some busy work to maintain itself, no matter how little it is, so spawning a large amount of micro VMs will bring more overhead than spawning a large amount of docker containers. On the other hand, virtual machines are obviously a lot better for security.
The performance overhead is probably not the biggest reason why you'd stick with Docker, though. The real kicker is that most existing services already run Docker containers and there are loads of resources out there to get you started. Firecracker is relatively new and unsupported. There's no technical reason why Firecracker is a better use case for an RPi, but I think people running RPi servers are on average less experienced and will likely benefit from the Docker knowledge base out there compared to, say, people renting a VPS somewhere, or people building full server machines.
Curious if Firecracker would be better about accessing some of the RPi hardware. I run node-red on RPi via docker and had to do some weird hacks to access certain pins with any reliability (it may have been a permissions issue, it's been a while and whatever i did has been solid since).
One thing I will say about docker is the packaging is very convenient. I have one container for node-red, one for InfluxDb and one for Grafana. I didn't have to deal with anything, just fire them up right out of docker hub and point at each other. Obviously a bit more scrutiny would be recommended for anything important but the overall experience was quite simple.
Another possible use case for Firecracker would be ones where folks are wiring up software-defined radios to RPi. (E.g. ADSB reflectors for FlightRadar24 and the like) Firecracker might be a more robust mechanism to ensure that the stream processing from the radio has sufficient capacity and priority vs whatever other cruft is happening on the box.
One of the reasons that I prefer containers over VMs which rarely gets discussed is that Docker doesn't make me configure SSH, process managers, logging, monitoring, etc. Further, as much as I dislike the standard Docker image build tooling, it's generally much easier than building and managing VM images. I think AWS Fargate gets it pretty correct by using Firecracker to spin up distinct VMs for every Kubernetes pod or ECS task (basically units of 1+ collocated containers). I would like to see more support for this in Kubernetes offerings for bare metal and other cloud providers.
As an aside, there's also krustlet--an experimental project that uses Kubernetes to schedule WASM programs (instead of containers) which are less versatile but which are much lighter-weight and more secure than containers. I'm really excited for WASM orchestrators like this.
EDIT: Wow, I really didn't expect this to be controversial. I'm really curious about what people are objecting to with this?
Depends what do you want to manage in your infra. I also like to manage most of my infra so I usually use VMs, packaging (Packer) and custom deployment tools (Ansible) but for development workflows, tests, environment replication sometimes I use Docker. These are not interchangeable entirely.
Fair enough. I was talking specifically about cloud environments. If you want to manage more things yourself, of course VMs are the way to go, but if you are looking to save yourself work I’m of the opinion that various container platforms are a very good option, not least of all because they let dev teams own more of the workload and thus less dependency on an ops team (and thus less coordination with the ops team, allowing the dev teams to move faster).
Depending on what you are referring to exactly. I do not think that could be that much runtime performance difference between those, boot time is a different question.
I can't comment on performance, but on a different note it seems to be generally accepted that 'real' container boot time is slow - fly.io mention this too [0]. To add some numbers:
- 40ms for nsjail run an isolated command and exit [1]
- 150[2]-250ms to boot a firecracker microvm
- ~450ms for docker startup [3]
There are probably very good reasons for the difference (e.g. docker has layered filesystems to set up), but the default experience makes a difference.
We've related the timing we've seen on our platform, and the fact that Firecracker works surprisingly well with the performance envelope we need, but I wouldn't go so far as to say Fly has benchmarked Docker vs. Firecracker. We're just relating our experiences. Security isolation is dispositive for us; we'd be using Firecracker even if it slowed us down.
Generally speaking containers replicate networking && storage layers so are definitely much slower than a vm on public cloud, not to mention they generally live on vms.
However, in this context (being on a rpi) it probably wouldn't be the case.
Now, as for the firecracker discussion: Firecracker trades a faster boot time for a slower runtime as evidenced by this issue here:
Citation very needed, given that Firecracker is a VM and Docker runs binaries on bare metal. Maybe faster startup, but I'd want to see testing methods and data before I believe even that.
Whatever happened to the containerd shim for Firecracker? Once complete, it would allow us to make Firecracker an implementation detail like using Docker/crio or containerd for running Pods.
I just read about people running Docker on a RP and thought, wouldn't Firecracker be better suited for this?