I'm hoping that this is the start of a trend away from the overly complex, overly engineered, big ball of mud that is Kubernetes towards simpler and easier deployment/hosting/container runtimes that are easier for smaller and large teams alike to manage.
Don't get me started on how "devops" has become a meme. I want to deploy services and features. I wouldn't want to spend my full time dealing with Kubernetes and containers and all manner of complexity.
I think cloud providers have much better models generally (e.g. Azure App Service). Setup pipeline, deploy, done. Containers are an optional thing.
What I'm saying is, I don't think it's fair on anyone that essentially the only choices are 1) run servers and VM's yourself and manage deploying to them 2) kubernetes 3) cloud providers.
> Kubernetes is complex because the reality of what you’re dealing with is complex.
Passing off complexity instead of reducing it isn't a net-positive though. The purpose of frameworks/libraries/etc is to (considerably) reduce complexity. In Kubernetes, you've replaced a group of specialist engineers with just a different group of specialist engineers.
What's complicated is to operate a Kubernetes cluster, nothing forces you to do that, major cloud provider all do it pretty well.
As for OP post lot of things don't make sense, I think the only valid reasons is about isolation which k8s is not very good at for now ( multi tenancy ). The rest is interesting... let them build there own solution to maintain and operate and see how it goes for them in 2 years.
Things that I found strange:
- Complexity "understanding how Kubernetes works and why it was implemented this way is hard"
So instead they're building their own tools? If you have the knowledge to build a similar platform, extanding and understanding k8s is easy.
- Global and multi-zone deployments "Implementing multi-zone with Kubernetes requires deploying a full cluster per zone, with a dedicated control plane for each data center." good luck with a single control plane controling all of you regions, one little problem will bring down all your servers. You should always have isolated regions. Nothing prevent you from building some sort of orchestration around multi region deployments.
- Scalability: Kubernetes is known to have limits in terms of the number of nodes in a cluster.
Yes there are limits, but looking at your startup I bet that 5000 nodes for a single cluster is more than what you need in a single region.
https://kubernetes.io/docs/setup/best-practices/cluster-larg...
Not sure about the other clouds, but GCP is just atrocious for this with both of their offerings.
App Engine, seems to be no longer developed, and is considered “old” technology, it works, but its hard to do more enterprisy stuff, like virtual private cloud (VPC) with it. For example you can configure it to be able to access a VPC, but you cannot put it “inside” the VPC so others services can interact with it.
Cloudrun seems to be the new hot thing, and it works too, seems to be made using kubernetes itself, so promises future development and integration. However there is a tiny flaw in the system, berried in a fine print in an obscure doc page. Cloudrun severely throttles cpus on any containers that are not currently processing an http request. So you cannot build anything that is long running - kafka consumers, batch processes etc. What you are supposed to do is delve head first into the whole of GCPs ecosystem and embrace their queues, message brokers, and event sourcing systems (as they’re all http request based), woe to the ones that have to interact with tech thats outside of it.
I'm currently testing our (GCP) solution to the CPU throttling you've highlighted. I've been using Vault[1] as my test case, and so far so good. Be on the look out for early sign up if you're interested.
Is Anthos an abstraction on top of Kubernetes then? Kind of hard to understand with all the marketing around it. E.g. it runs k8n, alongside other google tech on your hardware?
If that is the case maybe its going in the wrong direction from the general sentiment in this thread - trying to solve the problem of too many abstractions by adding another abstraction. I think I’d prefer tech that removes turtles from the stack rather than adding fresh ones.
Wonder how long it will take to have “Kipos - a tech to manage your complex Anthos deployments” :-D
NixOS with nixops is one such option (on bare metal or VMs).
You need to climb up the Nix learning curve a bit, but obtain an overall vastly less complex system than Kubernetes with its millions of lines of code and various levels of OS and networking encapsulation. It is possible for a single human to read and understand all code involved in a nixops deployment in a few months of side-time reading. It is possible to debug operational or configuration issues easily, with standard tools, and standard Linux systems understanding.
You manage your entire infrastructure, including all software and how to build it, declaratively. If you use cloud VMs, nixops will also create them for you.
It is not as powerful as Kubernetes; for example you currently don't get fully automated rolling deployments that roll back if your health checks start failing; you need to do this yourself e.g. running `--include machine1` through `--include machine3` to do a rolling deployment. However, this is often fine, as most small to medium teams do human-triggered deployment (and ensure that it works as expected) anyway.
I found that this works very well for small to medium teams that want to spend as little ops overhead as possible, while still understanding their stack entirely.
I heard about NixOS years ago and thought that an “immutable” Linux dist is an intriguing idea. Game changer if it actually works in practice. But I never properly experimented with it.
Are there any good learning resources for using NixOS in an “infrastructure as code” scenario, or should I just RTFM?
I've only written 2 tutorials in there so far doing basic things. In the future and as time permits, I'd like to write about how my org implemented various tasks with NixOps, such as rolling deployments, highly available database setups with automatic fallbacks, distributed file systems, how you can patch any software in your stack, how to write reliable systemd services with correct dependencies, and so on.
Kubernetes is complex ok but I disagree on cloud providers offering better alternatives, especially Azure. Azure is bug ridden and support depends on what type of price tier you are in. The thing with Cloud providers is when their services go down or change you are on the hook for supporting your app, not them. At least Kubernetes (or Nomad) lets you mange your own dumpster fire around which you can build safer deployment workflows.
> I'm hoping that this is the start of a trend away from the overly complex, overly engineered, big ball of mud that is Kubernetes towards simpler and easier deployment/hosting/container runtimes that are easier for smaller and large teams alike to manage.
But is it easier to manage? Is it simpler?
So they went from kubernetes to
- Control Pane running in
- Container running on
- Firecracker with Kuma running on
- Nomad with CoreOS running on
- Some servers
No idea what half of those words are, how they all are ocnfigured, interact with each other, and fail in spectacular ways.
To be fair many of these words would be needed for running a bare metal kube cluster.
- Some servers: you need servers to run a bare metal cluster.
- CoreOS: This is just a minimal OS that's designed to be PXE booted. Think of it as a docker container for a bare metal machine.
- Kuma: Multi-cluster mesh network. This makes it easier to write services that talk to `foobar:8080` and have it automatically fail over to other pods in other regions. I wouldn't use kuma, there's a few other options, but this is very helpful. Good service meshs will allow you to do Canary deployments. If you are multi-cluster than this sort of reliability is probably important to you.
But, an important part of this is that you don't need these if you're ok with running on another cloud's infrastructure.
In my experience I haven't bought into anything like that. The most I've done is paid for a hosted DB which is a complexity I could offload to many different companies.
I haven't used any services that have lock in, I've been able to ship services efficiently and meet deadlines, and have an abstracted compute layer.
I've also multiple times spun up demo environments with docker-compose on a bare metal machine to demo clients without networking.
Kube abstracts the compute and load balancing capabilities to a bare level. Everything else can be implemented within kube for at-cost prices.
> Kube abstracts the compute and load balancing capabilities to a bare level.
> Everything else can be implemented within kube for at-cost prices.
Until you need something more than a bare service with nothing in it: anything from messaging to databases to storage.
Companies don't usually run a bare-bones service in a kubernetes cluster. It needs to connect somewhere and do stuff. For example, we use BigQuery on GCP. Any idea how it "can be implemented within kube for at-cost prices"?
Don't use services that lock you in. Instead use a layer that abstracts that. In the past I think I have heard of people using Presto for similar operations and that can be self hosted. I would never build a company on top of tech I can only get from one place. Regardless of the layer in the stack.
If it's abstracted behind an agreed upon industry standard protocol (MySQL, pg, mongo connections or S3/NFS storage, etc)? Sure. If it's special sauce? No way.
My $Job-1 is migrating the stack I setup from Azure to GCP and it's been fairly painless from what I heard (only taking a few weeks to make an account, test, then move everything over).
> If it's abstracted behind an agreed upon industry standard protocol (MySQL, pg, mongo connections or S3/NFS storage, etc)?
This still doesn't answer the question of "can be implemented within kube for at-cost prices."
BigQuery is amazing at processing enormous amounts of data at fraction of time and cost it would take, say, MySQL to process the same data. How can I "abstract it away" and "implement it at cost with kube".
Even if it's purely standard stuff like PubSubs, or Apache Beam, there are value-added options that are hard to get from other providers (or implement yourself): anything from monitoring to management to scaling to...
I'm not to crazy about vendor lock-ins myself, but "abstract everything away and you can easily move over if needed" is largely a myth.
Wrt Azure App Service, thats basically just Kubernetes (well, AKS to be precise) under the hood anyway so it's really not an argument against the perceived complexity of Kubernetes. Which illustrates nicely that offloading the complexity of managing K8s clusters to cloud providers is a very valid way of doing things imo.
Having said that, I really don't like AAS, but that's a different story:)
> I'm hoping that this is the start of a trend away from the overly complex, overly engineered, big ball of mud that is Kubernetes
The main problem with k8s in the real world is that too often it's being deployed by people who want to fuck with k8s and have that on their CV, rather than people who want to provide something to make it easy to deploy workloads.
A huge amount of the k8s world and chatter is dominated by people who are doing the equivalent of, having been asked for a Linux server, providing Linux from Scratch, rather than a server running a well-known distro that people can just work on.
From personal experience I can only agree with their choice to pick Nomad. At the place where I work we have been running Nomad as our main container orchestrator for around 2.5 years now. It's rock solid, very easy to set-up and maintain and overall not too complex to understand in depth.
> Global and multi-zone deployments: User workloads on Koyeb need to be able to run in multiple zones. Kubernetes doesn't support multi-zone out of the box. Implementing multi-zone with Kubernetes requires deploying a full cluster per zone, with a dedicated control plane for each data center.
I’m having trouble understanding this. K8s worker nodes should be deployable across zones, regions etc. You can label the nodes with the zone id and use taints/tolerations to ensure workloads are deployed in specific zones/regions (if that’s what you want).
Yeah pretty sure it’s that bc k8s 100% supports multi-az on every major cloud provider out of the box. It also supports federation via kubefed but it needs a separate control plane in each region.
In theory nothing should be stopping you from deploying your etcd/master nodes in different regions but you’d probably need to tweak cloud resource provider to handle that and if one of regions is partitioned away from quorum those master become unavailable
Seems similar to fly.io, but fly.io seems better: uses WireGuard instead of service mesh, supports custom domains, supports any TCP or UDP service, provides volumes, etc. But that’s great seeing more options in that market!
I’m waiting the day someone discovers a simpler alternative to k8s. It will be like that day in which we realized that http verbs >>> Corba/rmi for web services.
I'm curious - when a user deploys their new code to your product, does that kick off a new Nomad job, or is that managed internally, kicking off a koyeb-managed Firecracker with Kuma for service discovery?
I maintain this small repo [1] and I have added our thread today to the section (#infrastructure). It's to learn "What/Why people move from this to that."
It's not an _awesome_-like repo but I've found it's useful to learn others' decisions. Feel free to keep them up-to-date (but if you know there is better place / resource, I'm happy to work with them too.) Thanks a lot.
Yes, we considered Consul as a service mesh but found that it was too strongly coupled with the network and task layer of Nomad.
We were looking for something highly customizable which wouldn't get into the way.
Also, the multi-tenancy features are paid features, which might have been difficult to sustain economically for us.
If things like multi-tenancy are required and the project (Consul) is open-source would adding that functionality yourselves not be in-scope given the amount of work you've already put into your setup?
Don't get me started on how "devops" has become a meme. I want to deploy services and features. I wouldn't want to spend my full time dealing with Kubernetes and containers and all manner of complexity.
I think cloud providers have much better models generally (e.g. Azure App Service). Setup pipeline, deploy, done. Containers are an optional thing.
What I'm saying is, I don't think it's fair on anyone that essentially the only choices are 1) run servers and VM's yourself and manage deploying to them 2) kubernetes 3) cloud providers.
There needs to be an option between them.