I wish their tech writers would give us more English and less jargon. I pondered these release notes and the documentation earlier today and I still can't puzzle out why "custom resource" is initialized as "CRD".
And the rest of this gibberish is just meaningless.
"""A resource is an endpoint in the Kubernetes API that stores a collection of API objects of a certain kind."""
OK, almost completely free of meaning ... then later:
"""A declarative API allows you to declare or specify the desired state of your resource and tries to keep the current state of Kubernetes objects in sync with the desired state."""
But wait, I thought the resource _was_ an API endpoint? The API allows me to declare the desired state of my API?
I know K8s is not Borg and I know a lot of xooglers hate Borg but at least the Borg documentation was concise and made sense.
A resource has an endpoint, i.e. a URL that represents its configuration and state. But a resource and its endpoint can be thought of as the same thing, in that the endpoint is the resource's canonical URI. Think REST/HATEOAS.
A resource is any object that Kubernetes can track. They include objects like services, pods, nodes, ingresses and so on. Together resources form an API.
A CRD is a Custom Resource Definition. It defines the schema and API of a custom resource (as opposed to a built-in one like "service" or "pod").
A resource is pure declarative data -- configuration and state commingled in one JSON document -- but can have behaviour associated with it.
(Confusingly, Kubernetes also uses the word "resource" to refer to compute resources like CPU and RAM. For that reason, you'll often see "object" used instead.)
It's not laziness at all. It's the project's Ubiquitous Language, which helps refer to concepts objectively and free from ambiguities. You're complaining because you are not familiar with Kubernetes, and thus you are not the target audience. Do keep in mind you're reading the release notes of a minor release.
With respect I disagree and my objection is that this verbiage is ambiguous and imprecise, the opposite of a ubiquitous domain language. Why does the “declarative API” allow one to “declare or specify”? Which is it? Is there meaning in the clause “or specify”, or is it a superfluous verbal tic? If the API is declarative why is it called a Definition? Declaration and definition are well-used terms of art in our industry but they are not synonymous. And what about “tries to keep them in sync”? Tried by what means? Are they eventually synchronized or not?
Ok but ingress controllers don't talk to the kube-controller-manager though, so I wouldn't lump them in as a standard object. In fact k8s doesn't even have a standard ingress controller object.
It's a OSS project, so if there is feedback filing an issue on the docs or submitting a PR is hugely beneficial. The docs team has a single full-time technical writer and the rest is contributions from the community.
I enjoy Kubernetes because it weeds me out from all the people who aren't used to actually having to read docs to understand the layered concepts of k8s as a networking model, of which there can be multiple opensource Implementations.
The opsys bros I know who are used to hacking together docker-composes that are more insecure than your grandmother's internet explorer browser in 30 seconds, slapping a json parsing logger on it and calling themselves geniuses fail miserably when they are left to design construct and optimise secure kubernetes clusters and elegantly port a variety of full stack web applications in a cost efficient manner.
It requires planning and design. Just because you haven't invested hundreds of hours into reading the docs and testing, and building k8s clusters for which the docs are incredibly useful doesn't mean you have to attack the verbage, just read the docs. If you can't do that then maybe your best investing your criticism into a domain of expertise you're actually knowledgeable and/or experienced in.
Also "kind" represents the type of k8s object, like service or deployment. It's not ambiguous at all, it just requires reading to the third paragraph of "general Kubernetes concepts" in the Kubernetes docs to understand it.
Both of your examples made sense to me, but when you take a step back that sentence doesn't really mean anything without context.
A "resource" === "endpoint" in the sense that the endpoint defines where you can define the configuration of a specific resource. They are being specifically vague I am assuming because of the concept of CRDs.
But without that sort of knowledge ("Hey, Kubernetes has a customizable API using custom resources!"), that sentence is rather hard to unpack.
edit: Thinking about it some more I suppose an endpoint is an overloaded term w.r.t. Pod IPs.
When you submit a CRD to Kubernetes you get a few things:
1. The CustomResourceDefinition itself, a document that tells Kubernetes "Hey, I have a kind of custom resource I want you to know about".
2. Instances of that custom resource that have been submitted to Kubernetes. "Submitted how?", you ask, which leads to:
3. When you submitted the CRD, Kubernetes automatically created HTTPS endpoints based on the values of the group, kind and version of the CRD. It accepts that custom resource at that endpoint.
What makes dealing with CRDs confusing, in my experience, is that "CRD" is used in two senses. One is the actual resource, the actual chunk of YAML submitted to Kubernetes that contains "kind: CustomResourceDefinition".
The second sense is to bundle together all the things that can be done with or flow from CRDs. Usually this is where you hear about controllers, operators and so on, but it will still be called "the Foo CRD", even though that's like referring to a 3-tier application as "the Foo table schema".
We're moving away from Kubernetes because managing even a simple implementation is a full-time job. Obviously Kubernetes is meant for enterprises, but I've seen seed round/series A startups with a small engineering team using it.
We were at a crossroad. Either we hire an SRE that has extensive kubernetes experience for $130K/year or we move to a managed platform until we actually need those capabilities.
We're happy with the move and the pressure relief is tremendous. Now we can focus on features and not whether or not deploying is using ENV vars from our local machines and causing the system to crash.
Why not just used managed Kubernetes? Every cloud vendor has it, even Tier 2 operators like DigitalOcean.
Running Kubernetes is vastly different for using it to run your apps. You really shouldn’t do the former unless you have to be on-Prem or have some other need.
"Managed Kubernetes" really runs the spectrum between "one step above just installing it yourself on a bunch of VMs" and "I spend 1% of my time managing anything below the product." Each cloud provider exists somewhere different on this spectrum, with none of them being in quite the same location, and some of them have multiple different products which exist at different points.
For example: AWS is among the most bare-bones. EKS is just a managed control plane; coming from GKE, you might click "create an cluster" then be very confused how there are no options for, say, instance size, or how many... because you have to do that all yourself. There are tools like eksctl or Rancher which can help with this, but ultimately, you're managing those instances. You're doing capacity planning (you think kube would be a great pick to integrate with spot fleets because of its ability to schedule and move workloads to a new instance when one goes down? have fun setting it up, hope you like ops work.). You're doing auto-scaling (and that ASG? its not going to know about your pod resource requests, so you either need some very smart manual coordination between the two, or you need to set up cluster-autoscaler). You're setting up cluster metrics (definitely need metrics-server. not heapster, that was last year, metrics-server is this year. but how to visualize? do i host grafana in the cluster? then i need to worry about authn. cloudwatch really isn't made for these kinds of things... maybe I'll just give datadog a few thousand bucks.) Crap, 1.16 is out already? They only support 9 months of releases with security updates?! I feel like I just upgraded my nodes! Oh well, time to lose a day replicating this update across all of my environments.
I'd go on, but you get the point. There is nothing "managed" about EKS.
DigitalOcean is pretty similar to this (it does provision instances, but the tooling beyond that is barebones). Google Cloud/GKE is "more managed" in a few sense; the cloud dashboard provides some great management capabilities out-of-the-box, such that you may not need to reach for something like Datadog, and the autoscaler works really well without a lot of tinkering. There are still underlying instances, so you're worrying about ingress protection, OS hardening, OS upgrades, etc... but its not as bad as AWS. Not by a long shot.
The holy grail (for some companies) is really something like Azure AKS + Azure Container Instances. No instances to manage. Click a button for a kubernetes cluster. Schedule workloads. Get functional metrics, logging, tracing, dashboards out of the box. Don't worry about OS upgrades, hardening, autoscaling, upgrading the cluster, etc; we'll do it all for you, or at least make it one click to configure. That's the ideal situation. I haven't used AKS/ACI so I can't comment on whether Azure gets us there, but the idea is sound; even if its more expensive.
This sounds like an anti-Kube post, right? Wrong (its a Tide ad). The beautiful thing about Kubernetes is that it can span this spectrum. The same exact API surface can scale from a fully-managed abstract platform where you just say "take this git repo and run it" (see: Gitlab Auto-DevOps), all the way to powering millions of workloads across dozens of federated clusters at Fortune 500 companies.
But, to the OPs point: We're close to solving that right end of the spectrum, and a lot further away from the left end. We're getting there, but we're not there yet. There isn't enough abstracted management of these compute resources... yet. But there's enough money and desire for there to be that I know we'll get there.
>For example: AWS is among the most bare-bones. EKS is just a managed control plane; coming from GKE, you might click "create an cluster" then be very confused how there are no options for, say, instance size, or how many... because you have to do that all yourself. There are tools like eksctl or Rancher which can help with this, but ultimately, you're managing those instances. You're doing capacity planning (you think kube would be a great pick to integrate with spot fleets because of its ability to schedule and move workloads to a new instance when one goes down? have fun setting it up, hope you like ops work.). You're doing auto-scaling (and that ASG? its not going to know about your pod resource requests, so you either need some very smart manual coordination between the two, or you need to set up cluster-autoscaler). You're setting up cluster metrics (definitely need metrics-server. not heapster, that was last year, metrics-server is this year. but how to visualize? do i host grafana in the cluster? then i need to worry about authn. cloudwatch really isn't made for these kinds of things... maybe I'll just give datadog a few thousand bucks.) Crap, 1.16 is out already? They only support 9 months of releases with security updates?! I feel like I just upgraded my nodes! Oh well, time to lose a day replicating this update across all of my environments.
Haha - I had to chuckle on this. It really is this bad on EKS. My god, upgrades are a total joke.
As bad as it is you have to believe that AWS seems to want to ignore Kubernetes and lock you in with ECS.
They need to get EKS as easy to manage as GKE and they need to provide free control plane/masters like all the other managed k8s services. Hopefully we see something at Re:invent this year... at this point there is no reason to use EKS unless you are trapped into AWS. Which unfortunately is what AWS is counting on :(
GKE and AKS both require setting up instance types for node pool(s) but you don't need to worry about the OS and related issues on either. Both also support auto-scaling and auto-upgrades.
GKE is far more managed though with advanced features like global network, aliased IPs, global load-balancing, istio/traffic manager integration, private IPs, metrics-based-autoscaling, preemptible nodes, local SSDs, GPUs, TPUs, etc.
Azure Container Instances are nice, with the GCP corollary being Cloud Run using KNative. Both products are designed to quickly run a container image with a public endpoint, but ACI is not part of AKS. You might be thinking of the Azure Virtual Kubelet which lets you burst on-demand to ACI, but this is a very advanced use-case and not the normal cluster setup.
> The holy grail (for some companies) is really something like Azure AKS + Azure Container Instances. No instances to manage...
Google Cloud have been having that since before it's being called Google Cloud -- I mean Google App Engine. It has all those features, and with its "Standard" environment you don't even need to build docker image.
Cloud Run is even closer. App Engine Standard requires you to use the sandbox, App Engine Flex is tied to VMs, but Cloud Run makes containers serverless:
I can attest that Cloud Run has been great for me. I literally "ported" over some of my apps that were already running in containers by replacing references to my existing env PORT_NUM to GCR's PORT and it just worked. It's been a life changer. Now if it could run non-http workloads, I'd be able to shut down every one of my VMs.
AWS, GCP, and Azure have a huge portfolio of datacenters, infrastructure, compute, networking, storage, security, databases, analytics, AI/ML, and other managed services that are completely missing from DigitalOcean.
DO is tier 2 because it provides basic IaaS components along with S3/Spaces, managed databases, load balancing, and K8S as compared to providers like Linode which would be tier 3 and below.
I don't understand. You had a problem; you needed to run a distributed system with containers which Kubernetes is perfect for. Did your problem disappeared or did your requirement changed or did you change to a different container orchestrator tool?
I'm saying we didn't have a problem that Kubernetes solved. A lot of seed-round/Series A startups also are quite simple, but hopping onto Kubernetes because it's "enterprise".
Adopting any piece of technology just because it's a fad or otherwise trendy is rarely a good justification.
It sounds like the primary reason you're deciding to move away from it is because you don't face any of the problems that it's there to solve, rather than it being an operational burden.
To be fair, k8s imposes a large complexity load. So what it gives you has got to be worth the time your developers/devops/sysops will spend learning to work with it.
We're migrating to k8s, and it's fantastic for us, but we have a large complex system that we're moving into the cloud, which really benefits from k8s features - loving the horizontal pod autoscalers, especially when I can expose Kafka topic lag to them (via Prometheus) as a metric to scale certain apps on.
It's really not "enterprise". It's because you have a resilient platform that provides automatic deployments, rolling upgrades, failover, auto-repair, logging, monitoring, storage/stateful data management, and more out of the box with a simple declarative YAML configuration.
Kubernetes can run a simple container with 1 line if that's all you need, or you can scale up to several different services with a few more files, all the way a massive deployment with thousands of containers. How you use it is up to you, but you do need to read and understand the basics.
However there's absolutely no need to run Kubernetes yourself unless you have a serious reason to. If you must, I highly recommend using something like Rancher [1] that can install and manage the cluster for you, even on cloud providers.
"I had a bad experience with Kubernetes" isn't really relevant discussion on a post about a kubernetes release. It's already well known that kubernetes is used by companies all over that don't need it and don't benefit from it because of its popularity.
In my experience, Kubernetes is a lot of work to install and maintain. However, I've found managed offerings—especially ones like Azure that actually abstract away the complexity (Amazon exposes all the nitty-gritty, in my experience)—make it easy to use for even smaller shops.
Of course, not every workload benefits from the features k8s provides.
I agree on EKS,IMO is it an unfinished bare bones "managed" service. Upgrades of control plane and nodes is a royal PITA - very manual and error prone. In GKE it is literally 2 gcloud commands.
I think AWS hamstrings kubernetes still bc it is holding onto ECS. Hopefully they see the light soon, beef it up and make it equal to AKS/GKE. At least make masters/control plane free...ugh
did you ever looked at k3s? what is more difficult about k3s than managing ansible?
what about kubespray? it's basically k8s managed via ansible? what makes it so difficult?
When folk want managed k8s, they effectively wanted something akin to a "serverless" compute pool that they point the k8s API at and worry about nothing else - not underlying hosts, networking configuration, storage, etc. The ideal is how close it gets to that. K3s still leaves hardware management to you.
For work projects, I don't want to care about hardware—or "machines" of any sort, for that matter. Give me a ready-to-go platform with networking, storage, node management, etc. already configured. I don't even want to have to run scripts.
At home, I'm happy to play around with such things—and just might (I just purchased an old Dell server to take care of some home tasks and provide a sandbox environment). But at work I want to be as far away from the bare metal (physical machines or VMs) as possible.
Don’t do Kubernetes or any other container orchestration when you don’t have CI/CD in place yet. I really think this is meant to work in tandem and changing Kubernetes YAML on a developer laptop can blow up fast.
Actually it's meant for virtually no one, because very few people have the problems it solves. Like AutoScaling Groups, Serverless, and Cloud its self, it's a tool that solves a pain point in a specific domain. About 0.001% of the business world have that problem.
> We're happy with the move and the pressure relief is tremendous.
I'll bet it was. I've convinced businesses to go in the completely opposite direction to K8s. I've told them to developed a monolith before they start optimising into microservices. The startups that listened to me are still around and in round C. The others closed up shop ages ago (minus one of them) because they never got to market in time.
K8s is a tool. Docker is a tool. Cloud is a tool. Businesses have to utilise tools as efficiently as possible to get their solutions out the door if they're to survive. Using K8s from the ground up is a death sentence.
The "Kubernetes is only for Google-scale companies" meme needs to die. Kubernetes is a useful tool even if you have a single node.
As a case in point: I just set up a small new app on GKE. Because I'm experienced with Kubernetes, within a few minutes I had my app (a React front-end written in TypeScript and a backend written in Go) running and receiving HTTPS traffic. I entered just a handful of shell commands to get from an empty cluster to a functional one.
It's a cluster with a single node, no resilience. But this is genuinely useful, and a perfectly appropriate use case for Kubernetes. The alternative is the old way — VMs, perhaps with a sprinkling of Terraform and/or Salt/Ansible/Chef/Puppet, dealing with Linux, installing OS packages, controlling everything over SSH — or some high-level PaaS like AppEngine or Heroku.
While it's an example that shows that Kubernetes "scales down", I'm now also free to scale up. While today it's a small app with essentially zero traffic, when/if the need should arise, I can just expand the nodepool, add a horizontal pod autoscaler, let the cluster span multiple regions/zones, and so on, and I'll have something that can handle whatever is thrown at it.
My company has a ton of apps on multiple Kubernetes clusters, none of them very big. From my perspective, it's a huge win in operational simplicity over the "old" way. Docker itself is a benefit, but the true benefits come when you can treat your entire cluster as a virtualized resource and just throw containers and load balancers and persistent disks at it.
Agree totally. One win Kubernetes provides my current team has been the fact that it is the interface on top of a cloud with agnostic primitives, allowing for multi cloud scaling.
There are a lot of primitives, and a lot of things to monitor to ensure a running cluster, but the complexity of the system is IMO overblown in the popular mind. Isn't the core tenant of SRE to have the stability of the system automated?
Can't agree more. That said, Kubernetes still has way too much RAM/CPU overhead in a single node setup. It scales up, but it doesn't scale down. Hopefully, it can be solved. There is k3s, for example, that limits the overhead by replacing etcd with SQLite, removing old stuff, etc.
> It's a cluster with a single node, no resilience. But this is genuinely useful
Can you elaborate? Sounds like replacing one kind of server management overhead with a new kind otherwise.
It's replacing one thing with something else, but the management overhead on GKE is significantly less.
I've also managed a built-from-ground-up Kubernetes cluster. It's not rocket science. That said, I wouldn't do it in a company without a dedicated ops person.
> My company has a ton of apps on multiple Kubernetes clusters, none of them very big. From my perspective, it's a huge win in operational simplicity over the "old" way. Docker itself is a benefit, but the true benefits come when you can treat your entire cluster as a virtualized resource and just throw containers and load balancers and persistent disks at it.
And I agree. However...
I'm referring to the companies and people who don't think like you do and instead think, "We need to build everything as a microservice and have it orchestrated on K8s". I'm thinking of the people who have a monolith that's scaling fine, solving a problem, and generating income, but an executive has been sold on K8s and now wants to refactor it.
Your use case makes perfect sense, but very few people are thinking like that, sadly.
In my opinion if you're starting out with Docker and K8s, you're going down the right path, but only provided you're not starting with microservices.
Like you said it's a tool and like with most tools there is a balance between over engineering its use.
You gave me a heart attack. I thought they were deprecating Deployment, StatefulSet… But it's only the old API versions they are deprecating, not the features themselves…
I enjoy Kubernetes because it weeds me out from all the people who aren't used to actually having to read docs to understand the layered concepts of k8s as a networking model, of which there can me multiple opensource Implementations.
The opsys bros I know who are used to hacking together docker-composes that are more insecure than your grandmother's internet explorer browser in 30 seconds and calling themselves geniuses fail miserably when they are left to design construct and optimise secure kubernetes clusters and elegantly port a variety of full stack web applications in a coat efficient manner.
It requires planning and design. Just because you haven't invested hundreds of hours into reading the docs.and testing, and building k8s clusters for which the docs are incredibly useful...
Also "kind" represents the type of k8s object, like service or deployment. It's not ambiguous at all, it just requires reading to the third paragraph of "general Kubernetes concepts" in the Kubernetes docs to understand it.
An ecosystem has evolved around Borg. Custom hardware, kernel, schedulers, telemetry, atomic clocks, networking, security, management... have all evolved around Borg proper to meet the "enterprise" needs of one of the largest enterprises in the world. It has many niche (i.e. not generally useful) cababilites built to service hardware melting XXX megawatt applications like web indexing, gmail, colossus. Even if Kubernetes targeted this customer, Borg has a significant head start. At this point the Borg ecosystem is almost old enough to by cigarettes in Mountain View.
Assuming that Borg it's proprietary (IE, zero documentation, no community, no real user cases outside of Google), yes. As all Google projects, let's wait its complexity doesn't collapse under its own weight, leaving no options but to move to somewhere simpler, which isn't on the horizon.
That sounds like a technical question that was given a political answer. ("Kubernetes is so great we don't want to keep it all to ourselves here at Google, let's give it to an open source foundation!")
My biggest quarrel with kubernetes is that its API Reference often contains broken link to non-existing documentation page, usually pages that have been moved somewhere else (but who knows where?).
Because safe generalized PATCH operations require the client and server to agree on the immediately prior state and the order of application of patches to avoid creating “Frankenstates”. They’re not a good general purpose API building block.
Cartoon giraffes in general look very disturbing to me. Generally the length of the neck and legs are about the same but somehow every cartoon giraffe has amputated legs.
And the rest of this gibberish is just meaningless.
"""A resource is an endpoint in the Kubernetes API that stores a collection of API objects of a certain kind."""
OK, almost completely free of meaning ... then later:
"""A declarative API allows you to declare or specify the desired state of your resource and tries to keep the current state of Kubernetes objects in sync with the desired state."""
But wait, I thought the resource _was_ an API endpoint? The API allows me to declare the desired state of my API?
I know K8s is not Borg and I know a lot of xooglers hate Borg but at least the Borg documentation was concise and made sense.