Our goal is sub-second container starts (admittedly, we're not there yet), and with Kubernetes we'd have to create Pods, create Services, wait for the scheduler to update, etc. We didn't go down the rabbit hole of profiling where the slowness was, but it was clear that Kubernetes just wasn't built with the type of speed we wanted. We realized we'd have to contend with a lot of design decisions that were the right choice for the things Kubernetes optimizes for (replication, resiliency), but not the right choice for us (fast launches of ephemeral containers).
> But on other hand, I can't quite figure out why something would prevent, you, yourself, from running the service that hosts the VMs that hosts the containers on demand on Kubernetes.
I'm not sure I understand this part, I guess we could use Kubernetes operators to scale up the underlying compute resources and manage the containers ourselves? This adds a lot of complexity for our use case.
> But on other hand, I can't quite figure out why something would prevent, you, yourself, from running the service that hosts the VMs that hosts the containers on demand on Kubernetes.
I'm not sure I understand this part, I guess we could use Kubernetes operators to scale up the underlying compute resources and manage the containers ourselves? This adds a lot of complexity for our use case.