sadly it's not cheap to built aws on bare metal.

geggam · on June 25, 2018

And you think k8s is simple to run on bare metal ?

To care and maintain ?

Do you understand the kernel tuning and routing Google does to avoid the NAT hell you are going to run into when you manage k8s on your own ?

merb · on June 26, 2018

I actually run 3 small bare metal clusters in production.

Maintain is basically just a `kubeadm upgrade` on all nodes + reboot (easily scriptable), after some updates get announced. OS Upgrades are done through container-linux-update-operator.

NAT is not a problem if you are below or equal to 50 nodes. Not sure if you run into problems if there are more nodes, thought. Also BGP+metallb is quite good.

(P.S.: openstack uses NAT heavily as well. and IPVS for k8s should fix most problems if you are running into problems)