> Say you want to deploy two Python servers. One of them needs Python 3.4 and the other needs Python 3.5.
Honestly hilarious. The core value prop example is if you want to run two minor version different programming languages on one machine. In order to do that, you get to deploy and configure hundreds to thousands of lines of yaml, learn about at least 20 different abstraction jargon terms, and continue to spend all your time supporting this mess of infrastructure forever.
How many engineering teams adopt kubernetes because it's what everyone's doing, versus out of genuine well-specified need? I have no idea.
I use k8s at work, i know it has benefits, but it too often feels like using a bazooka to butter your toast. We don't deploy to millions of users around the globe, and we're all on one python version (for library compatibility, amongst other things). Docker is an annoying curse more than it's a boon. How much of this complexity is because python virtualenvs are confusing? How much would be solved if instead of "containers" we deployed static binaries (that were a couple hundred mb's larger apiece because they contained all their dependencies statically linked in... but who's counting). Idk. Autoscheduling can be nice, but can also be a footgun. To boot, k8s does not have sane defaults, everything's a potential footgun.
In 10 years we're going to have either paved over all of the complexity, and k8s will be as invisible to us then as linux (mostly) is today. Or, we'll have realized this is insanity, and have ditched k8s for some other thing that solves a similar niche.
edit: I realize this is a salty post. I don't mean to make anyone feel bad if they love to think about and use k8s. I appreciate and benefit from tutorial articles like this just as much as the next dev. It's just the nature of the beast at this point, I think.
It is comforting to know others feel this way, especially the opening paragraph. Adopting complexity to solve complexity blows my mind.
What’s ironic, too, is the complexity is so far abstract from the problem it’s solving (and the problems it then introduces) that troubleshooting becomes a nightmare.
In my experience, too; few pushing adoption/application of these technologies are system administrators, or even architects/designers - or are only so fresh out of uni (which i do not mean to suggest is bad..)
The accountability becomes really muddy, not even considering security boundaries and other multi-business-level disciplines these “solutions” impact.
I get it. There’s a place for these technologies. But almost every adoption I’ve experienced to date is more of a shiny-thing syndrome.
> I use k8s at work, i know it has benefits, but it too often feels like using a bazooka to butter your toast.
This is such a great description (I also for my job work exclusively on a platform based on Kubernetes). I ran K8s at home successfully for a while using K3s (such a great project/tool) to become more proficient. After a few months I found that there were so many features I didn't need for a homelab that the complexity wasn't worth it.
I feel like, at the moment, Kubernetes is awesome when you have two ingredients:
1. You go with a managed K8s offering (such as AWS EKS)
2. You have a team of engineers dedicated to the health of the K8s platform
It's pretty cool that a somewhat small platform team can wield the scaleability of a company like Google, BUT they need to be very good with K8s. And many workloads probably don't need that much scaleability.
it can be automated just like it was automatable before, you know have a REST API to do the automation, that's the big change wrt being on bare metal (but you had Puppet/Chef/Ansible for almost 15 years now). If you were in the cloud, you already had a REST API to do that before.
Its very different though, you don't tell it the steps to do, you tell it the world state, kubernetes figures out how to transition between changes between states and if something breaks it tries to fix it by going back to the state its supposed to be in. I think this immutable/indempotency aspect of kubernetes is a bit underappreciated, this can't possibly be compared with ansible.
Further, the workload /just moves/ to another node. In the old ways, the service was tied to the node it was on. If a disk failed, the service failed. Now, within moments, that service moves and the node can just go away. It’s almost magic. I think the reason k8s doesn’t make much sense in a home-lab setting is due to the limited nodes.
IMO and IME stateful apps on k8s are still a PITA with lot of corner cases that can have really bad consequences. But as time passes, it's getting better.
Puppet was/is declarative, or at least tried to be (unlike Ansible). Don't get me wrong, I completely understand tje differences with K8S and I'd rather accept "the inner loop" as the biggest differentiator.
Yes, with Ansible you would be able to set this up. But with Kubernetes it is basically free. Kubernetes will manage your workloads and if you set up your health checks correctly it will do this for you. Kubernetes is designed for such tasks.
However. I would rather comfortably recommend Ansible for a home lab, but not Kubernetes. For something like that it is too complex with too many abstraction layers and too many caveats.
the real "killer app" of "workload orchestration" is the active control loops, configured (somewhat naturally) declaratively.
Ansible and other configuration managers and provisioning tools - as far as I know - had no way to do this. (they had policy conformance checks and whatnot, but fundamentally they did not make it easy to get to the desired good state, they ran on-demand and had to rely on ad-hoc information gathering, which made them slooooow and brittle)
The big difference in my opinion is that your ad-hoc servers using static binaries, with all the outside stuff specific to your organization, are non-standard.
For some miraculous reason K8S is ubiquitous and everybody uses it. Standardization is a boon.
People complain about git using similar arguments, and yet having a vast majority of the tech world on it is a boon for tooling.
Both those technologies are excellent but take some time to master (you don't have to be an expert though).
I can get on a project using git, kubernetes and say RoR and understand it very quickly without help. It is well bounded. It takes a git account and a kubeconfig. All set.
A custom python codebase deployed on custom servers running in big enterprise-y network, not so much.
How much would be solved if instead of "containers" we deployed static binaries (that were a couple hundred mb's larger apiece because they contained all their dependencies statically linked in... but who's counting).
Isn't that what a container is though? (static linking here meaning everything inside the container and no need to provide & manage external dependencies).
The container format has the advantage of simply being a standard linux environment, the kind of thing you work with anyway and can host any process type or application.
As for k8s, of those 100s or 1000s of lines of yaml, there's probably only 20 or so that you're interested in for a basic deployment. Tools like kustomize or pulumi make templating simple.
K8s is pretty simple under the hood. As an exercise, I once set up a cluster by hand on a couple of vms using kubeadm. It all dropped into place for me, understanding what kubelet was doing. K8s seemed a lot more sane for me thereafter.
No, the abstraction of a container involves not only namespace isolation, but also access to a certain quantity of resources to process and exchange information within and across namespaces.
So, pointing out “extra tools” (kuztomize) to ease difficulties with yet a new abstraction (k8s) is not really the way. You are just convincing yourself that your tool of choice can really solve everything.
Ideally, standard builtin tools should be easier to use, understand, and be less buggy by means of better opiniated solutions, just like some programming languages work for people who never coded before.
A container only knows about what's going on inside it though. If you want to hook those resources up with peers in a decoupled way, that requires an abstraction on top, preferably declarative, or else you faced with a substantial amount of manual (and thus costly and error-prone) management.
We could argue the merits of yaml and I'll happily concede it's not ideal, but the complexity isn't going away and the tools to manage it are pretty standard.
In 10 years Google will be a fading memory (one hopes). Once they die, K8s dies.
K8s' fatal flaw was being too big for its britches. As individual pieces, none of the K8s services are useful. Nobody downloads and installs just one of the components for use in some other project. And for that reason, it will always be just one big fat ugly monolith of fake microservices.
The architecture that will come afterward will be the model we've had since the 90s: single components that do their job well, that can be combined with other components in a hobbled-together way, that together make up a robust system. The same thing that lets you build an architecture out of PostgreSQL, Ory, Nginx, Flask, React, Prometheus, ELK, etc: they're all separate, but they can all be combined the way you want, when you want. And they're all optional.
I can’t help but think this is already here: docker, docker-compose, and proprietary cloud products. You’re not important enough to need a load balancer, and if you are you can use paas elbs and native services. Only once your google scale does k8s make sense.
IMO this is wrong way to approach explaining this. Having Python 3.4 and 3.5 coexist is laughable example - yes. Whole post is written from perspective of a developer who has code he needs to deploy.
If one looks from operations/sysadmins perspective:
- I have a big server or 10 servers
- I don't want to install PHP/Python/.NET for each dev team on the server or get new server and maintain it separately for PHP teams and separate way for .NET teams
- I want to provide environment where devs can just drop a container so it enables them to have what they need without me checking off each install and each VM
I also don't think dev teams should run their own k8s clusters. If you have huge company and people running data centers you don't want them to deal with whatever flavor of back-end code these servers are running.
> How much would be solved if instead of "containers" we deployed static binaries
That's pretty much what happens in the Go space, and people are still deploying k8s like crazy there, seemingly failing to consider the trade-offs involved. It's people's full time jobs to manage all of this when it really isn't needed if your ops story was just a bit simpler.
I'm a developer, so for the most part I don't really care what the ops people are doing, whatever works well for them, right? But the way k8s works and because of the "dev-ops" stuff it's also pushed hard on developers. I've worked at companies where it was literally impossible to set up a local dev instance of our app and even running tests could be a challenge, because unless you're very careful k8s gets in to everything.
Kubernetes feels like the lower level engine that some great abstraction system sits on top of, this thing with great defaults, fault tolerances, self-repairing and developer-friendly interface, unfortunately the higher level abstraction doesn't exist. Kubernetes makes that all possible and its ingenious and revolutionary, its just that its a terrible terrible experience to actually use in practice.
We need like what pytorch was to tensorflow, its clear that google has completely lost the plot again with this insane unmanageable complexity that blows up in your face every 5 minutes.
Although most of the articles focus on the benefits of Kubernetes for work, where you usually have more options (e.g. AWS Fargate), I've found it extremely helpful in the case where you want to self-host stuff personally/"at home", but you also want above average resilience so you use multiple nodes (e.g. 3 raspberry pi).
You can probably DIY something or always run all the things on all 3 nodes to achieve the same effect, but the first time I saw a node going down and Kubernetes automatically deployed the affected pods to a healthy node, it felt like having AWS on-prem.
Also, doing things in a standard way each time means it's really easy to bootstrap new things, not always reinventing the wheel. Especially cron jobs, I always lost track of those before. Now it's really easy to get an overview. Secrets as well, having it built-in just saves so much "how should I do this again".
Plus base OS upgrades are now really uneventful, since besides k3s (single-binary Kubernetes distribution), there's nothing else installed on the host.
The complexity is of course higher, you need a load balancer to point to all your cluster IPs since we don't get one "for free" from the cloud, shared storage is still something I'm looking into what's the best way to do (NAS with NFS, minio, ...), you most likely need a private registry, and probably others I'm missing.
Nonetheless, since migrating from manually run docker images, I find it a lot more comfortable and even with less downtime (taking down a single node is a non-event now versus "everything is down").
I still believe Kubernetes is one of the best tools out there to build scalable systems. But I wish it was more opinionated and less leaky.
Every project I build feels like a Frankenetes made of many subsystems provided by different vendors that struggle to talk to each other.
The day Kubernetes provides curated bundles for common architectures that I can spin up with a few commands, I will be in love with it. For now I have a love/hate relationship with it.
Totally agree that k8s is great, and that it needs deeper coupling to specific common app patterns to be more useful to devs (as opposed to platform teams). IMO a new layer of abstraction above k8s is the right approach, for a variety of reasons. We’re trying to build it at withcoherence.com, please check out our early MVP!
My rule of thumb is to go with the project that has the biggest community/support out there UNLESS the other project provides a business critical feature I need.
And even then it feels like we can just wait a bit until that feature gets implemented in the one with the biggest community/support.
For a startup, I'm still trying to figure out why you would use k8s instead of Google App Engine or other managed PaaS. You never worry about the problems listed in this article. You focus on writing code, deploying it with minimal config and let Google handle all these problems. Can anyone explain?
Because k8s is simpler than those solutions if you're not deploying things manually through the ui.
If you try to do something like infrastructure as code across multiple environments, with containers orchestration, service discovery, autoscaling, easy troubleshooting, authorization, easy domain name resolution, secrets at the container level then trying to do those things with App Engine becomes a horrifying ordeal.
The next question is obviously do you even need those things? But basic k8s really is easy to setup. Spinning up a cluster, writing some deployments, services, and setting up an ingress takes < 6 hours (half that if you do this regularly that you have documentation on it) and is the same experience everytime - there is no, I am using GCP at home so I don't know how to do this with AWS here. The CI/CD for the git repo is also pretty much the same.
Moreover, it's actually good. Sometimes you have industry standards that are just painful to use but do anyways because it's popular. With k8s, it really is a well thought out, mature solution that's better than most alternatives. Managed services often have a whole layer of bullshit e.g. last night I was looking at using App engine for a node app and I can't use the standard environment because I am using the latest version of Node that's not supported yet - and it's just one more nuisance to deal with. With k8s, it's just compute resources, containers, done. Simple.
You don't need to re-write the whole CI/CD stack, you don't need to re-engineer the app as a 12 factor app if/when the thing takes off and you need a real solution.
If your lead engineer gets hit by a bus tomorrow the whole deployment is in a helm chart somewhere and you can hire generic kubernetes tech #158326 to take over that job at any time.
Being able to engineer the app correctly the first time around, plus guaranteed business continuity are pretty big selling points. I would not invest in any company that did not have kubernetes, or a similar orchestration platform underpinning their product. Eventually that engineer will leave and you do not want to be scrambling for their replacement in the middle of the big growth push.
K8s probably wouldn't make sense for a lot of startups -- it's designed to solve a lot of different problems, and you're probably not going to be interested in solving most of those.
On the other hand, once you know k8s, it means you don't have to learn the specifics of any cloud platform. For most of my career I've worked in places that can't use cloud providers. I honestly dread having to learn all the different GCP or AWS tools/apis at a future job. But now that I have spent a lot of time learning k8s, it's nice to know that I can build on any cloud platform in the future, or self-hosted servers, and reuse the standard k8s APIs that I use at this job.
If I was building a startup, I'd use whatever the majority of engineers were already familiar with -- k8s, aws, gcp, whatever.
> k8s instead of Google App Engine or other managed PaaS
K8s is also available as a managed solution with GKE Autopilot, which in general is a breeze to use.
> You never worry about the problems listed in this article.
You still have to. If you look at the whole list of "how-to" guides one of the language docs for GAE (e.g. [0]), you see that there are about as much topics to worry about than if you were deploying to K8s. They also pretty much 1:1 map to things in K8s, but in contrast are non-standardized across vendors.
> deploying it with minimal config
I think the article laid out quite well how for a minimal deployment, the configuration you need for K8s is also pretty minimal. Yes, it might be more than a one-liner for a GAE config, but most of it is boilerplate you can ignore.
* you have workloads that don’t mesh well with common cloud PaaS patterns
* you want to pick-and-choose your cloud dependencies
* you want to use a commodity tech-stack so that hiring, tooling and support are far more readily available.
All of this comes with the obvious caveat of: sure, if you aren’t having issues, or reaching the limits of whatever PaaS solution you’re currently using, then don’t swap unnecessarily, but that still means there’s a lot of useful features that K8s offers.
For a startup that can do everything from scratch on one cloud provider, there aren’t many benefits. Those may come later once lock-in and complexity become a problem.
Besides opinions; remember that every tool has its uses. You can always cut down a tree with a shotgun, but a saw would be more efficient. Kubernetes is a complex beast, obviously not suitable for every workload and environment. But IMHO it can add value to product(s) when applied right.
The biggest problem I see is that the headline is ambiguous: it can be interpreted as "how to use Kubernetes to solve common problems," or alternately "how to solve problems commonly encountered in Kubernetes".
It's firmly in the realm of incidental complexity, viz. solving problems you wouldn't have in the first place if dynamic shared object loading, ld.so, and glibc would do their job or pointless Linux distro diversity wouldn't exist and/or static linking be used instead. And Python's inexcusable package management story, and even more inexcusable decision to base distro package management systems (yum, dnf) on Python of all languages.
Funny, I know a decent amount about k8s these days, but I've never really used AWS. I think there's a lot of systems that try to solve the same problems: "define these services, keep them running, here's how to check if they're down, here's how to restart them, keep resource usage to this level." Systemd, kubernetes, and individual cloud providers all have their own way to describe these.
One of the upsides of k8s is that it gives you a general, platform-agnostic way to describe this, on AWS or GCP or self-hosted or whatever. The downside is that it's more flexible and therefore more abstract than any particular provider. The upside is that you only have to learn one set of concepts (in theory).
AWS ECS is basically a docker swarm cluster customised by AWS to include auto-scaling of service containers/nodes and generally integrating better with other AWS services.
k8s is that orchestration layer, but on anabolic steroids. Plus a standardised interface on any cloud platform you might need to run on.
With bigger muscles comes more complexity, more maintenance work etc. (was designed for orchestrating millions of containers, so way more complexity)
which is why everyone always moans about it. (including me)
It’s a very cool and very useful piece of tech when yielded correctly. Just don’t expect it to work like a “kitchen sink”.
Yes, although a slightly higher abstraction layer that can be implemented by those things. So one can theoretically describe a complex environment in a way that can be run on prem or by a variety of cloud vendors.
(don't @ me just to say it's more complicated than that)
> Setting up an IngressController is probably the job of a specialized Platform team who maintain your k8s cluster, or by the cloud platform you're using.
What? I thought this was a solution solving problems, not making ones which you should not solve...
The team whose main objective is driving customer value only truly cares about the Ingress resource such as defining their routes to expose their work to the customer.
Unless you want every team to take the burden of logging, tracing, auth, etc you centralize around a global edge network which is where one would define a ingress controller resource.
> Say you want to deploy two Python servers. One of them needs Python 3.4 and the other needs Python 3.5.
Honestly hilarious. The core value prop example is if you want to run two minor version different programming languages on one machine. In order to do that, you get to deploy and configure hundreds to thousands of lines of yaml, learn about at least 20 different abstraction jargon terms, and continue to spend all your time supporting this mess of infrastructure forever.
How many engineering teams adopt kubernetes because it's what everyone's doing, versus out of genuine well-specified need? I have no idea.
I use k8s at work, i know it has benefits, but it too often feels like using a bazooka to butter your toast. We don't deploy to millions of users around the globe, and we're all on one python version (for library compatibility, amongst other things). Docker is an annoying curse more than it's a boon. How much of this complexity is because python virtualenvs are confusing? How much would be solved if instead of "containers" we deployed static binaries (that were a couple hundred mb's larger apiece because they contained all their dependencies statically linked in... but who's counting). Idk. Autoscheduling can be nice, but can also be a footgun. To boot, k8s does not have sane defaults, everything's a potential footgun.
In 10 years we're going to have either paved over all of the complexity, and k8s will be as invisible to us then as linux (mostly) is today. Or, we'll have realized this is insanity, and have ditched k8s for some other thing that solves a similar niche.
edit: I realize this is a salty post. I don't mean to make anyone feel bad if they love to think about and use k8s. I appreciate and benefit from tutorial articles like this just as much as the next dev. It's just the nature of the beast at this point, I think.