Yep, we (Basecamp) have been bit by this with EKS too. By default the ALB Ingress Controller will put all cluster nodes into the target group and a request can hit kube-proxy in any AZ before being directed to the right place, causing a lot of inter-AZ traffic churn. It's mildly annoying.
(with alb-ingress-controller you can change target-type to IP to have traffic go direct to pods that will actually back a given service instead of via kube-proxy on a random node, but the big fault there is that if alb-ingress-controller gets hung up on something and you deploy, you can send traffic to nowhere because the target group still has old IPs for pods that don't exist).
It's so worth it, though. You may not realize it, but your Healthchecks don't work unless you are using target-type IP. In the standard mode, AWS healthchecks each kubernetes node, but not your pods themselves. It will keep sending traffic to pods that are unhealthy (such as in state: terminating) if it already has connections open through kube-proxy to them, and stop sending traffic to healthy pods if the connection to them was through a node it considers "unhealthy". I wrote at tool[1] to handle this at Houseparty before alb-ingress-controller supported this mode, and it gained us a whole nine as measured by the ALB.
The same failure mode actually exists with instances, though they typically rotate much more slowly. Monitor your ALB-ingress-controller and treat it as critical infrastructure, because it is.
Can you expand on this: "AWS healthchecks each kubernetes node, but not your pods themselves".
Are you talking about a keepalive connection to an unhealthy pod which is reused for multiple requests? So the failure modes are, if I understand you correctly, a) the ALB keeps sending requests through an established keep-alive HTTP connection which terminates in an unhealthy pod, but which it sees as healthy because the node is healthy and can route traffic to another, healthy pod, and b) the health of an established HTTP keepalive connection is perceived to be that of the node rather than the destination pod, so nodes which become unhealthy can cause the ALB to unnecessarily terminate a keepalive connection.
We had to switch to using target-type=instance because of issues with pods not being deregistered. I'd prefer to use target-type IP but it seemed like preventing 500s on rollouts required a bit of testing and tuning with a very specific approach. e.g. introducing a longish delay on pod termination with a lifecycle hook and using the pod readiness gate support recently added to alb-ingress-controller.
You've got it exactly right. Your problem of pods not being deregistered is a real problem, but also with a quick fix: The default "Deregistration delay" for ALBs is 300 seconds but for kubernetes pods the TerminationGracePeriod defaults to 60 seconds. This means that your load balancer keeps trying that pod for 4 whole minutes after it's been hard-shutdown.
ALB Ingress Controller didn't cause a hard outage, lack of monitoring on critical infrastructure caused a hard outage. The ALB Ingress Controller was already critical infrastructure beforehand, because the NodePort method still relies on it adding and removing instances from the targetgroup - It just takes longer for that to become an issue. The fact that it's hours instead of days hurts, but also makes it much easier to diagnose.
Though for anyone using Pulsar on EKS who need multi-az for disaster recovery for the time being could deploy in a single AZ and then use the geo-replication feature in Pulsar to copy over to another AZ.
Regarding the alb-ingress-controller blackholing traffic with an IP target type ingress, check out support for pod readiness gates: https://kubernetes-sigs.github.io/aws-alb-ingress-controller.... Assuming you're using a rolling deployment, old pods won't be stopped until the new ones are added to the ALB target group and are in a healthy state. This has eliminated the small number of ELB 500s we'd see on deploys.
My disappointment with Kubernetes as the next/current big thing is that it practically didn't change a thing for companies as a whole. You still need dedicated devops people / a dedicated team. There is so much to get wrong regarding security, reliability and performance. Most application developers stay away from k8 configuration and could not quickly replace a devops engineer. They are the wizards holding the power.
It's an incredible mess of artificial complexity. You finally understood kubernetes and can work with the manifest files? Oh, no, no, no, nobody does that. You need to at least learn helm and a dozen of other kubernetes ecosystem tools! Every company has a different setup for it.
* I have a GKE cluster running to learn Kubernetes. Once you understood the basics it's indeed a simple way to get applications up and running, but my points stand. I am far away from enterprise readiness.
My experience is the same with the caveat that it made life somewhat easier for the devops team (of course, I was in the devops team here). All the common devops tasks were much easier (scaling, health checks, etc). Unfortunately, the least common ones were harder and lacked documentation to boot at least at first, until we got our own documentation to refer to.
To add to that, I think there's a very specific way you must do things to get K8s "right" and reap benefits. Or maybe it's a question of scale. There were honestly times when I thought maybe we could've stuck with good old servers. When I left the team was trying to redo microservices---this time properly---by introducing event queues and pubsub and finally split that database.
Love that workplace but I'm honestly glad I didn't get to participate in that transition. Sounds like "artificial complexity" as you put it. Would love to hear if they pulled it off though or not, a project postmortem basically.
>You still need dedicated devops people / a dedicated team.
Why is this surprising? You always need people to manage stuff. Was there any point where Kubernetes was advertised as a way to NOT have sys admins/devops?
Kubernetes is made for the people that have to run your systems and keep them operational (ops/devops). It makes our lives a billion times easier on two fronts. Disclaimer, I am referring to a managed solution like GKE, I can imagine running k8s on your own hardware would be harder.
Uptime & Stability: autoscaling, health checks, replacing nodes, restarting containers. These are all things that an Ops person does not have to worry about during day to day operations.
Visibility: This is the biggest thing for me. People hate CLIs but kubectl and it's standard commands (get, describe, etc) make my life so much easier as it allows me to quickly figure out why something is breaking. A typical troubleshooting session goes like:
- What customer are you looking at? (kubectl get namespaces)
- What service is failing (kubectl get pods -> kubectl describe pod <pod name> -> kubectl logs -f <pod name> -c <container>
and the error is often quite obvious.
This visibility is powerful and is something you can easily treat your developers to do on their own. Even though Ops owns the systems they do not have to be a black box to the Developers that use them.
I'd like to hear more on troubleshooting with a managed k8s platform and how you diagnose such issues.
My organization is playing with EKS and while perhaps it is different from GKE I feel like I have minimal visibility into if the control plane is healthy. I've had several instances where all of my node groups have become unhealthy for various reasons and EKS failed to spin up new nodes to resolve the issue.
GKE is a magnitude better than the other managed kubernetes services.
Observability starts with logging and metrics. K8S already puts out lots of events on what's happening which you can use to debug, and there are many 3rd-party monitoring solutions if you need more detail.
If you don't have this with EKS then use something like LogDNA and NetData to see the logs and metrics yourself.
The features i mentioned, are solved 'out of the box'. I'm not seeing a huge stretch here at all.
Its costly and risky to run 100 VMs, maintaining them and keeping them up to date, monitoring them and knowing when they are no longer needed.
It is cost ineffective to have security audits on 100 VMs, maintaining access to them, auditing whats happening on them.
It is a ton easier to allow someone only access to one namespace, limited ingress domains (which get provisioned automatically) and allow them to only run non root containersl
The question is not what the actual benefit is (its clear and i mentioned it in my paragraphs before), the question is will kubernetes ever become so lightweight, stable, easy to use so that its feasable for normal people to run it on a 3-5 node cluster in small companies.
And please lets ignore all those small companies where people log into their VMs by hand and maintaining them by using snapshots and cloning existing VMs. Those small companies might and should just migrate to cloud managed services completly.
A tiny fraction of companies need to run 100+ VMs. It takes a lot of traffic to require that scale, unless you're just throwing money at infra to avoid optimizing your code (which can be cost effective).
A large chunk of companies should just be on something like SquareSpace or Shopify, another large chunk should be using off-the-shelf services like Azure's App Containers or Amazon's Elastic Beanstalk/Lightsail/whatever else they have now, and some internal services can easily run on a single VM with no orchestration at all.
All that stuff is really expensive at scale, but unless you're a large company, your engineering time is going to cost way more than just using whatever managed services your cloud offers.
> ...maintaining them and keeping them up to date, monitoring them and knowing when they are no longer needed.
This is not the only alternative to using Kubernetes. You can automate all of this without Kubernetes, or you can use a cloud provider's managed Kubernetes service. A lot of companies get by just fine with Heroku.
If you assume people are using practices from 2005 (or that cloud providers don't already provide a layer of abstraction on top of Kubernetes), of course Kubernetes looks better.
I did mention myself that small companies should just use cloud services right?
I did not say that everyone should migrate to kubernetes as far as i rmemeber but kubernetes to me is not a fad and it fixes real issues companies have.
I dunno, because of writeups like GitLabs? We're lucky GitLab is so transparent. It lets us see that their claims don't appear to always match the actual reality.
Here's an example.
Blog post says: "After transitioning each service, we enjoyed many benefits of using Kubernetes in production, including much faster and safer deploys of the application, scaling, and more efficient resource allocation"
Wrong. Actual analysis by the engineers of one of their migrated jobs (urgent-other sidekiq shard) says their cost went from $290/month when running in VMs to $700/month when running in Kubernetes. They tried to use auto-scaling but failed completely and ended up disabling it:
Kubernetes looks like a massive LOSS in the case of this service. The perceived scaling benefits didn't materialise at all, and their costs more than doubled. They also spent a lot of engineering time optimising startup time of this job: 7 PRs and complex writeups/testing were required. Just so they could try to auto-scale a job to reduce the hit of the hugely multiplied base costs. If not for Kubernetes that eng time might have been spent adding features to the product instead.
It solves really problems out out the box like load balancing, ingress, cert management, autoscaling, health checks, autorepair
Most of these problems are created by Kubernetes, like "ingress" which is a term only Kubernetes and low level networking uses, "cert management" which often isn't required in more traditional setups beyond provisioning web servers with the SSL files, "health checks" are a feature of any load balancer and the whole point of VMs is to avoid the need to repair hardware. Finally the demand auto-scaling as seen in this case is (a) possible without Kubernetes and (b) not actually working well anyway.
Frankly this set of bugs, writeups, PRs etc is quite scary. I worked with Borg at Google and saw how it was both a force multiplier but also a huge timesink in many ways. It made some complex things simple but also made some simple things absurdly complex. At one point I had a task to just serve a static website on Borg, it turned into a three month nightmare because the infrastructure was so fragile and Google-specific. A bunch of Linux VMs running Apache would have been far faster to set up and run.
> Actual analysis by the engineers of one of their migrated jobs (urgent-other sidekiq shard) says their cost went from $290/month when running in VMs to $700/month when running in Kubernetes. They tried to use auto-scaling but failed completely and ended up disabling it:
Note this is just one service out of many and has very low latency requirements, and due to our slow pod start times in this case it didn't make sense to auto-scale. We are auto-scaling for other services in K8s.
> says their cost went from $290/month when running in VMs to $700/month
These numbers represent a snapshot in time where we are overprovisioning, for this service for the migration. You are correct that cost benefit here remains to be seen for this service. We are doing things like isolating services into separate node pools which may not allow us to be as efficient.
Safer and faster deploys was a huge win for us, for this service and others. This of course is compared to our existing configuration using VMs and Chef to manage them.
Its really weird that your argument is based on just the cost.
I mean you mentioned yourself you worked at google right?
Perhaps you just haven't actually experienced the issues kubernetes is solving?
How often have you seen that certificates expired? I have seen that. Plenty of times. Its not an issue creating that lets encrypt cronjob, its still something you need to do right.
Security Updates? Have you seen how many companies run with old non updated VMs?
Disk full due to logs? Yes seeing this regularly.
Memory leak on a service and someone needs to restart it manually until someone else fixes the issue? Yes!
Requesting a VM, hardware, infrastructure, getting it and the whole lifecycle management of it in the backend? Its real.
Ansible Scripts, puppet or just bash scripts and a word document to tell you how this magic machine was set up? Yepp.
Kubernetes solves all those problems.
Your static website on borg, if it sill runs, has probably still a valid certificate, is running on a secure infrastructure, is equally configured on every instance and not weird on 1 of 6 servers and just runs.
A smart person taking responsibility for all of this, costs you much more then just a few hundred bucks a month. And you need that person. With Kubernetes, this person now can manage and operate much more servers under his/her fingertips better easier and more secure then if it would have been vms.
And in my personal experience: That shit runs more stable because that shit can restart and being recreated and it just solves a handfull of shitty memory or disk full issues.
I've been running my own Linux servers for about twenty years, so I'm not entirely unexperienced with these things.
Borg is/was great for running huge numbers of services at truly massive scale, when those services were developed entirely in house and done in the exact way it wanted services to be done. It was a terrible cost the moment you wanted to run anything third party or which wasn't written in that exact Google way, and the costs were especially high if you didn't need to handle huge traffic or data sizes.
Borg and Kubernetes don't auto-magically do sysadmin work. A ton of people work in infrastructure at Google. Software updates still need to be applied etc. In Kubernetes that means rebuilding Docker images (which in reality doesn't happen as the tools for this are poor, so you just have a lot of downlevel Ubuntu images floating around).
And worse, the whole K8s/Docker paradigm is totally backwards incompatible. At Google this didn't matter because the software stack evolved in parallel with the cluster management. Programs there expected local disk to be volatile, expected to be killed at a moment's notice, expected datacenters to appear and disappear like moles. They were written that way from scratch. But that came with a terrible price: it was basically impossible to use ordinary open source apps. You could import libraries and (carefully!) incorporate them into Borg-ized projects, but that was about it.
When this tech was reimplemented and thrown over the wall to the community, the cultural expectations that came with it didn't come with it. So I've seen situations like, "whoops, we deleted our companies private key because it was stored to local disk and then the Docker container was shut down, help!". This was even reported as a bug in the software! No, the bug is that your computer is randomly deleting critical files for no good reason, and normal software does not expect that to happen. How about the way in a Dockerfile you have to write 'apt-get update && apt-get upgrade' if you want a secure OS image when the container is rebuilt? If you put the two commands on separate lines it will appear to be working, right up until you start getting weird errors about missing files from Debian mirrors, because Docker assumes every single command you run is a pure function! And then we get to security.
Screwups seem to follow Kubernetes/Docker around like flies. The tech is complex and violates basic expectations programs have about how POSIX works. When it goes wrong, it leads to mistakes in production.
Now there have been a few trends over time:
1. Hardware has got a lot more powerful. It has been outpacing growth in the internet and economy. Many more businesses fit in a smaller number of machines than when Borg was designed (an era where 4-core systems were considered high end).
2. Cheap colo providers have been driving the cost of powerful VMs down to the ground.
3. Linux has got easier to administer.
These days setting up a bunch of Linux VMs that self-upgrade, run some services via systemd etc isn't difficult, and properly tuned such a setup should be able to serve a monster amount of traffic (watch out though for Azure, which seems to overcommit capacity pretty drastically and their VMs have very unstable performance).
Most businesses that are deploying Kubernetes today quite simply do not need a million machines. Even companies that give away complex services like GitLab, as we can see from this thread, they don't really need huge scalability. It's just nice to imagine that the business will experience explosive growth and that growth is now automated, but ironically, the effort to automate business infrastructure scaling takes away from the sort of efforts that actually grow the business.
As for the other benefits, you can write a systemd unit that is much simpler than Kubernetes configuration that will give you auto-restart, including if memory limits are hit, you can view the activity of a cluster easily using plain old SSH or something like Cockpit, it will handle log rotation for you out of the box, sandboxing likewise, and so on. And of course the unattended-upgrades package has existed for a while.
I agree you need someone to do admin work, whatever path you choose. Having used Kubernetes, and the system it is based on, and plain old Linux, my intuition is that the base cost of Kubernetes is too high for almost all its users. Too many ways to screw it up, too many ways for it to go wrong, too much time spent screwing around, and too expensive. If you become another Google or Facebook then sure, go for it. Otherwise, better avoided.
> Wrong. Actual analysis by the engineers of one of their migrated jobs (urgent-other sidekiq shard) says their cost went from $290/month when running in VMs to $700/month when running in Kubernetes.
This is only one of the shards of one service that we migrated, and we migrated many more.
if you are curious. I am happy to have a conversation with anyone about our experiences in more detail, not to convince anyone that k8s is the "one-true-way" (because I am still not convinced myself), but of the benefits this type of change can bring.
> They tried to use auto-scaling but failed completely and ended up disabling it
That issue is just one of many though. We disabled it at the time, but we have it enabled for a number of other services.
Regardless of how I personally feel about K8s, I have to say that the migration we are doing for GitLab.com is generating set of benefits that goes far beyond just moving to a new platform.
One of the largest benefits I've seen so far is that it was a great forcing function to resolve some long running architectural challenges, and is making us think more about how the application can run at a very large scale, without being at the very large scale. Things that we could get away before, like the issue you referenced there, we can't anymore.
Disclaimer: I am one of the people involved in this migration.
Technologies don't last long. Another new tech comes out that does the job better or more cheaply, and the old tech either dies or competes.
I think that K8s is much too complicated for medium-scale workloads, so as soon as something makes it easier than K8s to scale and manage workloads at that scale, K8s will be out of that space.
And thats why you have to stay on your toes and migrate away if it starts to make sense.
It doesn't make much sense to migrate to k8s if you don't have an issue. And it doesn't make sense to wait for the next thing to happen when you have an issue you need to fix now.
Kubernetes hype looks, feels and smells very similar to what was happening with OpenStack. Lavish conferences, every vendor out there announces "support for Kubernetes" in their products, evangelists and many many burnt engineers.
Do you have any argument for it or is it a generic statement independent of anything?
Because there have been other 'fads' still being strong today.
Kubernetes solves real problems which have been hard for a long time. Its the first thing you, as an infrastructure team, want to have to be able to provide your teams a manageable environment for yourself.
mesos, nomad, docker swarm and co.
Developers don't want a VM and you can't manage and maintain VMs if someone else is doing something with them.
Cloud native is here to stay; It will affect and already affects tools, applications and architecture.
> Developers don't want a VM and you can't manage and maintain VMs if someone else is doing something with them.
I want a VM because I want my dev environment to exactly replicate the production environment, down to the kernel.
It also means that if I leave my employer I can just delete my VM on my personal machine.
We've also not had any issues with maintaining production VMs, our release pipeline packages up our code into a debian package and them creates a VM image from that. This is then deployed across all regions. If you need to roll back just deploy the previous VM image version. GCP handles killing old instances and starting new instances for you.
Also i'm not able to determine if your setup would still be much better (faster, easier to maintain etc.) if you would set completly on containers instead of a VM as i'm not aware of your workload at all.
That is a whole different discussion. If by application container you mean Tomcat then maybe. But if you mean Weblogic or Oc4j or WebSphere then hell no :-)
Then maybe you agree that Kubernetes brought some advantages that until recently only Java devs enjoyed to the rest of the world (PHP, Python etc)?
Kubernetes clearly, obviously and rather openly IS the application server of the current decade. I am saying this as a dev who has seen up close: OpenShift, Kubernetes, Tomcat and enterprise WebSphere.
Now you can have all this complexity even in other languages than Java.
For many simple applications, an aws autoscaling group and application load balancer is as good as k8s and far simpler.
If you're running microservices (to the point where you have so many small services that having an asg per service), than the above strategy does waste resources compared to k8s I guess... But you'd need a lot of microservices to justify the significant overhead k8s has.
But now you have to manage those VM's, the OS lifecycle, different distributions, application deployment, configuration and dependencies. And we're back at containers. Or you have to go towards lambda.
For K8s you need to manage the vm's/os as well but that part is mostly trivial as it's highly focussed on immutable/disposable infrastructure.
Not OP, and don't even have an alternative, but I find it disappointing that kubernetes basically seems to be the only answer for 'running containers in production'. Docker and its tooling is pretty development focused, and Kubernetes is probably too complex for 95% of use cases, which makes me think there's gotta be something in the middle ground that reaps the benefits of containers, wraps them with good tooling, reproducibility, reliability, and observability, and avoids the complexity of k8s.
AWS Fargate has worked really well for me. And only required an afternoon to get working. I've spent weeks learning kubernetes and still don't know enough to not hit rough edges.
I agree, but man so many places are going in the opposite direction… I got lucky with my current gig that I got to implement load balancing/autoscaling with Fabric, ~1000 lines of python in a daemon using boto3, nginx and disk images…
systemd will happily health check a service, restart it when it fails and so on. The basics don't need much.
Auto-scaling is something I'm suspicious of. I saw many experiments with that at Google when I worked there, under the name of elasticity. They found the same thing GitLab found here: it's really hard and the expected savings often don't materialise. Even in 2012 their elasticity projects were far more advanced than what Kubernetes or AWS provide.
Most cloud auto-scaling efforts appear to be driven by the high base costs charged by the cloud providers in the first place. I've seen a few cases where simply moving VMs to cheaper colo providers saves more money than auto-scaling VMs or using managed Kubernetes in AWS/Azure.
I found AWS's autoscaling offering be better than GCP's, in that you could use custom CloudWatch metrics as a basis for scaling logic rather than just the standard metrics (CPU usage, latency, etc)
GCP does offer custom metrics, but I couldn't figure out how to tie them to the LB scaling logic.
> This 'k8s' fad is the open source version of what you are using
That would be AWS EKS. ECS is very different beast. Wins big in infra-as-a-code (CFN). And simpler, fully integrated, and you get full AWS support, not the case with EKS or hand rolled kube.
> .. monitoring is a fad ..
We did some pretty fancy things in monitoring. ELK and more. And yes, it uses some useful bits of CW.
Another happy ECS user here. You get most of the benefits of Kubernetes, without a lot of its complexity and maintenance. Yes, it’s a proprietary service and all, but that’s not really an issue. The applications are already containerized and could be moved to any other vendor/tool with relatively low effort if you ever needed to.
> We found that with an application that increases its memory utilization over time, low requests (which reserves memory for each pod) and a generous hard limit on utilization was a recipe for node saturation and a high rate of evictions. To adjust for this we eventually decided to use higher requests and lower limit which took pressure off of the nodes and allowed pods to be recycled without putting too much pressure on the node.
Kubernetes works at its most efficient when you scale by adding more Pods, rather than scaling within a Pod. Requests and limits should be equal to each other, and a HorizontalPodAutoscaler (or KEDA Scales object) should be used to add more Pods as utilization increases. Using Cluster Autoscaler then makes sure that your cluster has sufficient Nodes to schedule the Pods. Then you just need to set up alerting to tell you if the autoscaling reaches minimums (indicating that you should consider lowering the minimum, to improve efficiency) or maximums (indicating that you should consider increasing the maximums), and you can leave things well enough alone, unless the speed at which scaling occurs is insufficiently slow for your use case.
There are some complains that we can share, but honestly they are not really that fun to read. We are still early in our transition and one major complaint I have is about helm and lack of flexibility of it and operator being very flexible but a huge time sink.
I think a lot of complaints we would usually have are reduced due to the fact that we use managed K8s with GKE.
All other complaints I can think of are related to our application and some architecture decisions we made early on, but that won't be interesting to anyone but people at GitLab.
I do have only very minimal xp in k8s, as I have tried to use it only once (and never again). To me it seem like mOst of the comments are about the unusable and unmaintainable yaml configuration files. This seems like exactly the issue I had a few years ago, when I was cursing, because I could not find a spec or docs for the yaml stuff in all the pages and pages of seemingly important documentation and thus could not figure out how to transform a docker only deployment into k8s deployment. I also remember having to set up helm, according to whatever tutorial or official guide I was following. It was all a distant nightmare, that I do not wish to repeat.
There's definitely a need for an opinionated layer over the top, with much simpler configuration. Probably more than one, with different use cases. Nobody should have to wrangle that much yaml.
This. I'm currently trying to figure this mess out and all I want is a reference for the different objects and what parameters they take, or what the hell an "apiVersion" is and which ones are available. Instead their "documentation" is a bunch of unhelpful guides that never show examples of what I need to know. How this has reached widespread adoption is beyond me.
I find that the best place to have an actual documentation for writing yaml files is the API Reference[0] and falling back to the "regular" documentation for more detailed explanations about specific topics. The reference lists all properties and type for everything that you can shove in your yaml files.
We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?
`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster.
The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything.
What I'd like is to generate serialized yaml files ... you know ... with code. Using some typed programming language, to be able to build any abstractions required for the job, and get some checks, compiler errors, and even ability to have asserts, unit-tests and so on. Instead, I have to dig YAML with a pickaxe in the YAML-mine without any technology to assist me.
K8s was born to solve the same problem Borg solved or simply design in such a way to not have that problem.
One of them is the config files hell.
Borgcfg files ranks #3 in terms of line count across all of Google. (Borgcfg language is similar to Jsonnet, helm)
In terms of the amount of power wielded buy a single CLI, `borgcfg`, this little program that has less than 50k lines of nontesting code, stands at a peak dwarfs almost anything else inside Google.
I declared in 2015 that k8s will eventually enjoy the same borgcfg problem. And I was advocating code driving application management approach as you suggested, not the configuration as code (behind the infrastructure as code facade).
Well, it seems I am still on the track to be on the correct camp...
The yaml monster in k8s is quite unpleasant. It's always been surprising to me how we somehow ended up with executable BUILD files but totally static deploy files.
@mpvl is one of my most admired Googler. I promised to contribute to the Cue project, but haven't found time (typical startup employee situation, but it seems there is no trouble to find time surfing hacker news...).
I don't know if I think Helm is garbage but I feel like it's almost always more trouble than its worth. Most charts can be boiled down to just a couple resources and instead of sifting through its weirdo templates you might as well just look at the actual resource configs. It's almost always just a single Deployment or similar + RBAC.
I've actually been using Terraform to manage Kubernetes resources at my latest job, along with literally everything else. I don't even use alb-ingress-controller or external-dns or any of that. I just have Terraform make target groups from the service resources. It breaks a lot less.
But Helm barely adds any boilerplate over that 'single deployment or similar + RBAC'? Just a Chart.yaml (minimal of which is very small) and adding the option to parameterise or templatise the resources.
I don't know why you wouldn't, we don't say 'I don't know if .deb/.rpm/PKGINFO/etc. is worth it, most packages are just a binary'.
For Helm to add LoC you'd have to all but not use it, and even then it'd be about 3LoC. Setup is no harder, with N kubectl applies becoming 1 helm install.
I also started using terraform (directly instead of just triggering helm) - are you using kubernetes tf resources directly though, or the helm provider?
Most helm charts I have seen and used have a deployment yaml or statefulset or daemonset plus a service yaml, optional ingress yaml, sidecar containers for metrics, and several others I'm forgetting at the moment. And I would say most as in 90% of the charts on helm hub.
If you're doing more than storing config values in YAML, you're playing with fire. I've really grown to dislike the trend in "developing by descriptor". It's great when it works and impossible to debug when anything goes wrong. And also, not easily testable.
This is why I use Terraform for my kubernetes objects. Yes, it is a DSL and not perfect but it has many advantages over helm such as:
- State. I cannot express how useful having the desired and actual state of a system is. It makes it so much easier to determined what has changed in between terraform applies which in turn makes tracking down problems easier.
- Providers. Being able to chain resources together from other systems is very powerful. As an example, I can:
1) create a database with a random name
2) create a user/password for that database
3) save the data above as a kubernetes secret
4) pass that secret into my kubernetes container for the service to use.
And you can easily change database to be anything such as a storage bucket, dns entry, twilio account, stripe account, etc.
Now try and do the exact same thing with helm!
- Providers Part 2: You can write your own providers to talk to your OWN services. I have done this and written up some go code to talk to our internal APIs when setting up new customers. It is very easy to create your own terraform resources to do this.
Carvel/k14s has been very nice for the usecase I had. Use ytt to generate the yams, using a yaml aware variant of skylark and kapp to manage the actual atomic deployment, delete, etc.
It’s very simple and worked quite well for me.
Disclaimer: work at VMware but on a different team (it’s an OS tool built by some of our engineers, which we use internally)
Could you explain why helm is garbage? I think it suits its purpose rather well without being too complex. You can essentially "plug-in" different types of resources rather easily. Especially in v3 now that you don't need to install Tiller and can avoid setting those cluster permission requirements.
Have you tried some Kubernetes api libraries? You can generate and configure resources with [python kubernetes-client](https://github.com/kubernetes-client/python) without much trouble. Personally I prefer editing them as JSON instead of python objects, but it isn't too bad.
1. YAML string templating makes it very easy to get indentation and/or quotation wrong, and the error messages can easily end up pretty far from the actual errors. Structured data should be generated with structured templating.
2. "Values" aren't typechecked or cleaned.
3. Easy to end up in a state where a failed deploy leaves you with a mess to clean up by hand.
4. No good way to preview what a deploy will change.
5. Weird interactions when resources are edited manually (especially back in Helm 2, but still a thing).
6. No good way to migrate objects into a Helm chart without deleting and recreating them.
7. Tons of repetitive boilerplate in each chart to customize basic settings (like replica counts).
It's a typical Go solution, in all the wrong ways.
It's not going to solve all your problems but dhall can fix your first few gripes. I've been using it for several months and it's an excellent way to write configuration imo.
Yeah, I have used Nix to generate them in the past, which worked pretty great too. But Helm does, admittedly, solve a real problem: garbage collecting old resources when they're deleted from the repo. I just wish we could have something much simpler that only did that...
`kubectl apply --prune` should nominally do this. Irritatingly (I acknowledge I'm almost as responsible as anyone else for doing something about this), it's had this disclaimer on it for quite some time now:
> Alpha Disclaimer: the --prune functionality is not yet complete. Do not use unless you are aware of what the current state is. See ⟨https://issues.k8s.io/34274⟩.
I haven't used it in anger, so I can't add any disclaimer or otherwise of my own.
kpt is the recent Google-ordained (AFAICT) solution to this problem, but is ot yet at v1.
You could also resolve this yourself by either:
* versioning with labels and deleting all resources with labels indicating older versions
As helm charts become used by more people and more complicated, exposing more of the underlying config, they just turn into a set of yaml files with as much or more complexity as the thing they are trying to replace. Configuration with intelligence that allows arbitrary overrides of any of the underlying configuration is important in order to meet all use cases. Without that, helm will only be useful for a strict subset of use cases and eventually you will outgrow the chart, or the chart will grow in complexity until its worthless.
We've found Kustomize, or just straight up writing the deployments ourselves the best approach.
The actual spec for a Deployment/Daemon Set/Stateful Set/CRD is usually super straightforward once you get the Kubernetes terminology, and most of the issues I've had with Helm have boiled down to "oh they haven't parametised the one config I need to change"
I always have some configs that helm hasn't parameterized. But that's not a problem because I always fetch my charts from helm hub into my repo. So I just add any parameters I need.
Helm's stated goal is to be something of a package manager for apps in k8s, but this is fundamentally unworkable as shown by... Helm. It's hard to describe just how unworkable this idea is.
Let's start with an example, you want to install an app (let's say Jira) and a DB backend of your choice, postgres or mysql.
The first step where this all falls down is, it may or may not support your preferred DB. Sure, Jira does, but does the chart?
Assuming it does support your preferred backend, maybe it depends on the chart for the db you picked. If it does, it's going to install it for you, hopefully with best practices, almost certainly not according to your corporate security policy. This is also a problem if say, you have a db you want to use already, prefer to use an operator for managing your db, or use a db outside k8s.
You got lucky, and it supports your DB just the way you want it. Next question, do you want an HA Jira? Often, this part is done so differently that HA Jira and Single host Jira are straight up different charts.
Do you want your DB to be HA? Unfortunately, the chart Jira chart author picked to depend on is the non-HA one. Guess you're out of luck.
Maybe you want to add redis caching? Nginx frontend/ingress? Want to terminate TLS at the app host and not ingress? How do you integrate it with your cert management system?
We haven't even looked at the config, where you have to do everything in a variables.yaml file which is never documented as well as the actual thing it's configuring on your behalf, and is not always done in a sensible manner.
Hopefully it's clear from this that as a user, helm isn't going to work for you, because just as there's no such thing as the average person, there's no such thing as an average deployment. Even a basic one is filled with one off variations for every user that a public chart needs to support.
As a developer, helm is unworkable because you're templating yaml inside yaml. This isn't too bad if you're just tweaking a few things on an otherwise plain chart, but a public chart, that naively hopes to support all the possible configurations? Your otherwise simple chart is now 5-10x longer from all the templating options. Have fun supporting it and adding the new features you'll inevitably need to add and support.
As a counterpoint to all this, kustomize gets a lot right. I don't mean that kustomize is perfect, or even good, but I've found like k8s itself, it understands that the problem space is complex and to try and hide that complexity leads to a lesser product that is more complex because of leaky abstractions.
Kustomize acts as a glue layer for your manifests, so instead of some giant morass of charts and dependencies none of which work for you, you're expected to find a suitable chart for each piece yourself and compose them with Kustomize.
Going through the same example again as a user:
Your vendor has provided a couple basic manifests for you to consume, maybe even only one, because they're expecting you to supply your own DB. Since they only need to supply the Jira part, instead of having an HA chart and a Single node manifest, They just give you one manifest with a stateful set. Or maybe they give you two. One as a deployment and one as a Stateful set. The stateful set might also have an upgrade policy configured for you.
Since the vendor punted on the DB, you can do whatever you like here. You'll have to supply a config map with your db config to the Jira deployment, but that's okay, it's easy to override the default one in the manifest with kustomize. You are now free to use a cloud managed DB, an operator managed one, or just pull the stock manifest for your preferred DB.
Want to terminate TLS in your app again? Easy enough, Cert Manager will provide the certs for you, and supply them as secrets, ready to consume in your app, you just need to tell it where to look.
So now you have all the parts of your Jira deployment configured just how you like, but they're all separate. Maybe you just edited the stock manifests to get your changes in the way you like. Dang, you've created a problem when the vendor updates the manifest, as now you need to merge in changes every time you update it. That seems like a huge hassle. What if you could just keep your changes in a patch and apply that to a stock manifest? Then it's easy to diff upgrades of the stock manifest and see what changed, and it's easy to see what you care about in your patch.
All of this seems like it's getting kind of unwieldy, maybe we can make it easier. We'll have a single kustomization.yaml, and it'll be really structured. In it, you can list all the paths to folders of manifests, or individual files, or git repos/branches/subpaths. We'll also specify a set of patches that look like partial manifests to apply to these base manifests to make it clear what is the base and what goes on top. Then finally, for common things like image versions and namespaces, we'll expose them directly so you don't need to patch everything. We can do that because we're using standard manifests that can be parsed and modified.
That is kustomize, and as awkward as it is, it's just trying to make it clear what you're applying, and what your one customization from stock is in a maintainable way. It does a better job of 'package management' by not managing packages. This is pretty similar to how Linux package managers work. If you install wordpress on your server, it's going to install php, but it might not install apache or nginx, as everyone wants something different. It's definitely not going to install a DB for you. You as the admin have to decide what you want and tell each part about the other.
Thanks for the detailed post. Really helps newcomers like myself.
One of the benefits as a new user of k8s is the ability to grab a helm chart to get me most of the way with something like ELK. I want to go the way of Kustomize but can't seem to find the same with it?
If you want an ELK stack, you should look into the operators provided by Elastic [1]. All you need to do is write a very small manifest or two for each thing you want operated. I feel like this is a better solution to 'I want an ELK cluster' than a helm chart because it solves more problems without leaking.
It does away with the notion that you need serialized yaml files, although in its current incarnation it can produce them, and instead runs programs which generates collections of named (and hierarchically defined) resources which can then be "applied" as a batch.
It uses lazy values to generate a directed graph to determine the order for creating and tearing down resources as well as safely propagate configuration outputs and secrets.
I generally agree. I'm using kustomize right now, but I'm also using helm charts via a kustomize plugin called chartInflator. It allows us to re-use existing upstream boilerplate while still using kustomize. Though I've not been entirely happy with kustomize either, and I'm thinking I might begin to investigate jsonnet for some things.
Honestly, I am not a fan of "configuration languages". Why do we need another bunch of languages. Were the existing languages not good enough? There's already shitload of them.
The only place where I find it useful is when the config is provider by an untrusted party. But even then scheme, lua would probably fit the bill.
Not strong feelings about it though, maybe I'm wrong.
All I want is a library for my favorite language, where I can easily output bunch of `yaml` files to feed to k8s. I guess pulumi is kind of like that, but I never had a chance to try it.
I don't know about the others, but at least Dhall provides features that are literally impossible to get from a general purpose programming language, like semantic hashing, totality, and such.
My experience with using general purpose languages for things like configuration or build systems has not been great. You always seem to end up with building a bunch of abstractions on top of that language, which culminates in another in-house configuration/build DSL, for which no public info is available.
It is also very hard to prevent people from using the escape hatch of the general purpose language to "fix a problem quickly", while introducing unpredictable side effects.
Scons (using Python) and Gradle (using Groovy / Kotlin) are 2 cases where I have seen this go very wrong, very quickly. I now strongly prefer Maven, even with its warts and XML verbosity.
On the other hand, configuration languages also often suck, the abomination that is HCL (e.g used in Terraform) comes to mind (especially in its early days). Turns out it is really hard to make a good language, and a bad configuration language is even worse than using a general purpose language.
I do like Dhall though: sufficiently powerful so you never have to repeat yourself, while also preventing you from doing fancy stuff you will later regret, all with a very readable syntax, and the type system helps to catch typos / mistakes early. I now use it to generate all config files for tools that consume json/yaml, and often for other text based config files as well (where you can use it as a powerful type-safe template engine)
Helm is probably the worst influence I've experienced on the kubernetes ecosystem. It takes something that could have been really good (metaconfig) and makes it illegable, complex, and impossible to debug.
Kustomize, too, doesn't get at the root of all the needs I've found myself having.
A metaconfig language like Jsonnet, Dhall, or Cuelang seems to be where it's at. If you're using jsonnet I can also highly recommend kubecfg [0] as it's a tool that "just works" and is built at the perfect level of abstraction: "just give me something and I'll find all the kubernetes objects inside of it". It's remarkably simple and forgiving and it makes it possible to describe even the most complex application configs + kube configs in a single language. You can also take it a step further and describe your CI stages (pre-merge, release, builds, etc), kube yaml, and infrastructure provisioning (terraform) all in one language.
> What are the benefits of using helm?
The main benefit is redistribution. Since everyone uses helm it's easy to give someone a chart because they know what to do with it. That's pretty much the only upside I've found.
Dhall continues to a be a glimmer of hope in this space. I agree that helm caused the entire space to distort. There is so little reusability in charts.
Dhall seems to be the start of reversing this, but I hesitate to mention it since the ergonomics are still rough.
If YAML is the assembly language of k8s, Dhall looks like what you would write the compiler in.
IMO because Dhall is unique syntax to the vast majority of developers who have never done anything with Haskell or Elm or whatever, it’s a non starter for large case use. We use Starlark, where the syntax is familiar to anyone familiar with python.
I like your analogy though. For us, the assembly language is using the k8s API directly in Golang. The “compiler” is the golang Starlark interpreter extended with our own config API, like you would implement in Dhall. It’s just in this case, you can implement it in Golang, which has much much better tooling than Dhall does. A typed compiler, debugger, IDE, unit tests.... so much easier to develop and maintain.
Yea, anything that allows you to think in a higher-level than raw kube objects will save you a lot of time and heartache. For instance, if your application needs a redis cluster in helm you have to mess with subcharts/other headaches. In jsonnet it becomes `redis = import "redis"; redis.Cluster("my-app-name") { nodes: 10 }` or something similar.
Given that the Dhall type system supports things that the other configuration languages do not, isn't it very hard to migrate to for a project like k8s?
I don't see how they could support something in parallel to Dhall, and still enable all the features Dhall provides.
Dhall does have many features that can still "compile down" to YAML or JSON (or some others, too). Depending on this target language, some Dhall features are unavailable.
I don't think OP meant Kubernetes is changing to Dhall, but that we'll see something like Dhall on top of Kubernetes YAML/JSON that catches on.
For internal k8s config at our org we built an config DSL using Starlark. The golang Starlark interpreter is super easy to use and extend. Starlark is familiar to every developer in our org because we are a Python shop. The tooling then spits out k8s YAML.
Essentially the config language implementation would be the same logic that a helm chart would do, but you’re writing the logic in Go versus a text templating engine. You can easily unit test parts and rely on the compiler to catch basic mistakes. Way better than templating YAML.
We also provide escape hatches so people can easily patch the resources before they get serialized to YAML. People can use that to customize our standard deployment config however they want.
So far this has worked very well and been extremely easy to maintain.
I actually use and prefer make + kustomize -- unfortunately the version of kustomize that is bundled with kubectl now is a bit behind on features sometimes but still it does "just enough" and I manage the rest with make.
Redistribution is important (and harder with a simple setup like make + kustomize) but I think obscuring the resources you need to run an application (custom or otherwise) is actually a bad thing for everyone in the ecosystem. If you're going to run something in your cluster, intimate knowledge of the resources required, the ways it changes your system is important.
- I already use Helm to install open source software; there are already so many Charts available. So that makes it easy to also use Helm for my own apps (I use it just for templating though)
- Helm templating language is more flexible, but that can also bite you since it can become complex. With Kustomize, changing a single environment variable value in a Deployment is quite some work compared to Helm (I don’t like JSON patching)
> With Kustomize, changing a single environment variable value in a Deployment is quite some work compared to Helm (I don’t like JSON patching)
Config maps, enough said. I have a CD pipeline that writes environment variables to files, Kustomize eats them up and creates a configmap, environment variables in the deployment reference the config map, done.
Kapp as a layer on top of Kustomize is showing some progress. Ultimately Kustomize is like every other resource bundled with k8s, it’s a building block for higher level tooling.
Blog author here - we are using Helm for our application because of how much we have already invested in our public Helm chart.
We are not sold on using Helm however for K8s in general and are currently evaluating Tanka for some of our monitoring infrastructure. One of our engineers wrote up some notes on that here if you are curious https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issu...
Helm is a wannabe package manager, designed for people who really like writing yaml instead of executable code. K8s doesn't really have a fully built out "apps" resource, so that starts the whole process down a gravelly slope of pain... with initial care and if your team need to groom their k8s configs, you can go a long way with helm before the pain starts. A particular problem with helm is that it isn't really a fully fledged programming language, but a glue laid over the golang templating system.
I use helm as little as possible. Usually only enough to template out a single file with the different bits of k8s infra stuck together with `---`.
If I was doing some relatively serious work on custom services with minimal needs to conform to best^H^H^H^Hpopular practices, my service's configuration would be a neatly packaged Java program (its libraries for K8s direct API access are good).
There are different approaches to handling ops tasks; I favor the ones that are rooted in writing executable code over the ones with yaml.
Some bright chappie will probably write a Common Lisp library derivation library and k8s will be obvious and simple with that interface, but I'm not that bright - all I can see is the potential.
Helm is good for importing other components (eg Redis).
I find it pretty tedious for authoring my own Yaml specs; any time you need to do some sort of composition you have to write templates, and then mess with indentation operators, because significant white space. Templating yaml is a pretty bad idea in my experience.
As of Helm3 the most egregious part (Tiller) is gone, so it’s more recommendable.
Scaffold lets you write (roughly) native k8s yaml files and layer them together. IDEs know the schema for this yaml at this point, so you get code completion on all your input files. This breaks in Helm. Scaffold is good for simple apps, and falls down a bit if you need to dynamically configure your specs (eg render a parameterized manifest for each review app). The programming model for scaffold is a bit hard to grok, based on my experience of ramping up a team on it. Not rocket science just a bit unintuitive at first.
Helm is amazing. If you are migrating legacy applications that rely on filesystem config, you can publish that config in a helm chart that applications require as sub charts. Gives you the ability to do centralized, versioned config management for legacy workloads going to the cloud
Declarative paradigms get a bad rap but I really just believe its people who missing imperative/functional programmimg. Theres a reason the infra space has been using declarative syntax for years now
The declarative functionality comes from K8s not helm. (Of course K8s has imperative tooling too but that’s not the point here).
The main reason folks dislike helm are 2 fold:
1) using string templating on a space formatted language is just messy and ugly for no good reason. If you don’t do anything terribly complicated it’s not too bad but quickly can be pretty fragile and unreadable.
2) golang(which I like generally) but the templating is messy and adds to the hard to read later problem
Personally find helm’s opportunity for code reuse to be restrictive and folks just end up cut and pasting a lot of declarations all over which is ... so terrible. But the first bullet above is pretty a fundamental design flaw.
Those are fair infrastructure reasons, but not necessarily cost-effective. There are always competing perspectives. What does the business want. What do the developers want. What does the infrastructure team want. When power is centralized in one area, decisions can be made that overlook those competing purposes.
I see containers being adopted without openly discussing the options and long-term consequences.
Was anyone else surprised by the timeline of this post? I was under the impression that Gitlab had been fully on K8S for years now, given the hard push of their K8S offering 18 months ago.
It has been possible to run GitLab the application on K8s for some time now for the majority of self-managed deployments. There were some limitations however, for running GitLab on K8s at the very large scale of GitLab.com that we have been working through in the last year.
> August 2019 when we migrated the GitLab Container Registry to Kubernetes, the first service to move. Though this was a critical and high traffic service, it was a good choice for the first migration because it is a stateless application with only a few external dependencies
I was under the impression that porting a large stateless service was something that wasn't too bad to do with K8S? What limitations prevented this from going ahead earlier?
Yeah, this is why we decided to migrate it first about a year ago, and we probably could have done it sooner than that. Other services were not yet ready to migrate until recently due to some blocking issues which was a factor in the decision.
Yep, we (Basecamp) have been bit by this with EKS too. By default the ALB Ingress Controller will put all cluster nodes into the target group and a request can hit kube-proxy in any AZ before being directed to the right place, causing a lot of inter-AZ traffic churn. It's mildly annoying.
(with alb-ingress-controller you can change target-type to IP to have traffic go direct to pods that will actually back a given service instead of via kube-proxy on a random node, but the big fault there is that if alb-ingress-controller gets hung up on something and you deploy, you can send traffic to nowhere because the target group still has old IPs for pods that don't exist).