We came from Ansible managed deployments of vanilla docker with nginx as single node ingress with another load balancer on top of that.
Worked fine, but HA for containers that are only allowed to exist once in the stack was one thing that caused us headaches.
Then, we had a workshop for Rancher RKE. Looked promising at the start, but operating it became a headache as we didn't have enough people in the project team to maintain it. Certificates expiring was an issue and the fact that you actually kinda had to baby-sit the cluster was a turn off.
We killed the switch to kubernetes and moved back to Ansible + nginX + docker.
In the meantime we were toying around with Docker Swarm for smaller scale deployments and inhouse infrastructure. We didn't find anything to not like and are currently moving into that direction.
2. Graphical Insights into the Cluster / Debugging -> Portainer
3. Ingress: Treafik together with tecnativa/docker-socket-proxy so that traefik does not have to run on the managers
4. Container Autoscaling: did not need it yet for our internal installations as well as our customer deployments on bare metal, but we would go for a solution based on prometheus metrics, similar to https://github.com/UnclePhil/ascaler
5. Hardware Autoscaling: We would build a custom script for this based on prometheus that automatically orders servers of Hetzner using their hcloud-cli
We came from Ansible managed deployments of vanilla docker with nginx as single node ingress with another load balancer on top of that.
Worked fine, but HA for containers that are only allowed to exist once in the stack was one thing that caused us headaches.
Then, we had a workshop for Rancher RKE. Looked promising at the start, but operating it became a headache as we didn't have enough people in the project team to maintain it. Certificates expiring was an issue and the fact that you actually kinda had to baby-sit the cluster was a turn off.
We killed the switch to kubernetes and moved back to Ansible + nginX + docker.
In the meantime we were toying around with Docker Swarm for smaller scale deployments and inhouse infrastructure. We didn't find anything to not like and are currently moving into that direction.
How we do things in Swarm:
1. Monitoring using an updated Swarmprom stack (https://github.com/neuroforgede/swarmsible/tree/master/envir...)
2. Graphical Insights into the Cluster / Debugging -> Portainer
3. Ingress: Treafik together with tecnativa/docker-socket-proxy so that traefik does not have to run on the managers
4. Container Autoscaling: did not need it yet for our internal installations as well as our customer deployments on bare metal, but we would go for a solution based on prometheus metrics, similar to https://github.com/UnclePhil/ascaler
5. Hardware Autoscaling: We would build a custom script for this based on prometheus that automatically orders servers of Hetzner using their hcloud-cli
6. Volumes: Hetzner Cloud Plugin, see https://github.com/costela/docker-volume-hetzner - Looking forward to CSI support though.
7. Load Balancer + SSL: in front of the Swarm using our Cloud Provider
Reasons that we would dabble in k8s again:
1. A lot of projects are k8s only (see OpenFaaS for example)
2. Finer grained control for User permissions
3. Service Mesh to introduce service accounts without requiring to go through a custom proxy