On: BPF, XDP, Packet Filters and UDP

[tptacek]

You could describe our job as "taking Dockerfiles from customers and running them globally"; the way we actually "run" Docker images is to convert them to root filesystems and run them inside Firecracker. Firecracker is the core of our stack.

[k__ → tptacek]

I find it rather curious that the cloud-native crowd tries to sell us containers, but cloud providers themselves use VMs.

Like, if not even AWS, Cloudflare, and fly.io use containers, how can K8s be native in any way?

I mean, that makes even Lambda, which runs on Firecracker VMs, more native than K8s.

[mrkurt → k__]

Most organizations don't have the multi tenant problem we do, and end up just using Docker when they do containers.

But I also think it's fair to call "Firecracker VMs" containers. Most of what those people are talking about is application packaging and deployment, not necessarily what actually runs.

For what it's worth, I am also cynical about "cloud native".

[michaeldwan → k__]

Fwiw we run as many services as we can on our own platform. Mission critical systems like our registry, api, redis servers, and much more are all running as fly apps in firecracker.

[riyakhanna1983 → tptacek]

Why not just run Docker containers natively?

[mrkurt → riyakhanna1983]

We have a whole post about workload isolation that answers this: https://fly.io/blog/sandboxing-and-workload-isolation/

The tldr is: Docker containers don't offer enough isolation for multi tenant systems.

They're also very slow to boot, compared to a Firecracker VM.

[riyakhanna1983 → mrkurt]

But does FC not incur high I/O overhead at runtime?

[mrkurt → riyakhanna1983]

The flippant answer is "it doesn't really matter, safety usually wins over performance".

But also, we run Firecrackers on our own physical servers and performance is quite good (even network + disk performance).