I wonder what the need for tools such as this or other "Kubernetes-by-example" type pages tell us about the complexity of configuring Kubernetes resources.
Do we need a better layer of abstraction, i.e. better adoption and tighter integration for something like kustomize? Have we fucked up completely with Kubernetes due to it being outrageously complicated for simple tasks? How we redesign this to be simpler? Is the complexity even a problem for the target audience?
I've no idea. I just know I'm a kubernetes admin and I can't write a deployment yaml without googling or copy/pasting.
I think of Kubernetes as "the new Operating System", and these complex resources as fiddling with initscripts, fstab, /etc/interfaces, and so on. Writing an operator is like writing your own initscript.
I wouldn't be surprised if we eventually see new abstractions for "you just want a plain 'ol deployment with CI/CD du jour, a persistent volume claim, and a service ingress, just like 90% of all other CRUD webapps? Sure, here's a simple resource for that."
I think we'll start seeing a move towards more "opinionated" tools, just to outsource some of the decision making. No sense learning how to write your own pipelines if you can find a tool that says "we're gonna deploy every time you make a git tag and run `mvn package`, you figure the rest out".
Helm has been doing this since the early days. 99% of our charts are created using:
* `helm create` to get the default scaffold
* modify a handful of entries in the generated values file
* done!
Only thing is the default helm chart starter does not allow for autoconfiguring of volumes, and since we're porting a lot of stateful apps to kubernetes we just modified the default starter to include that capability.
Of course it would be nice to not have to maintain a bunch of different virtually identical templates.
Helm is an example of what I mean by "pusing against the boundaries."
We tried using helm, but we kept running into limitations, and the complexity of trying to customize charts that didn't work right ended up being worse than the original problem.
Currently, we're using kustomize, plus a thought out manifest hierarchy, which is working out better.
As someone attempting to port a docker-compose application to a Helm chart I'd love to hear more about the resources you used. As someone working with a very simple application I've found the processes to be more difficult than I anticipated.
Kubernetes will be far more complex than docker compose. It'll let you do much, much more also.
Some potentially useful tips:
* One docker container usually maps to one pod, which is created by a deployment. You can put multiple containers in a single pod, but this couples them together tightly.
* Use services to assign hostnames to pods
* Have pods communicate to each other using the service names. This works the same as putting them in the same docker network.
* Volumes depend on your cloud provider or if you're running on bare metal. In the cloud it's easy, you just request one and it gets created on demand and backed by a cloud disk.
* If you're using volumes, you probably want to use the Recreate updatePolicy for your deployment. This will ensure the old pod is shutdown before creating a new one. Which is necessary to work with block volumes.
When using helm start with a `helm create CHARTNAME` and take a look at what it generated for you. You'll get some heavily templated yaml that if you're lucky you will barely have to touch.
And the tower of ever more obscure abstractions just keeps on growing. Until one of these fragile layers starts to malfunction and the whole thing comes tumbling down and nobody has a clue how to fix it all up.
People have attempted simpler abstractions and it has always failed. CloudFoundry, Mesos+Marathon, Heroku were all simple at first and none of them revolutionized the industry.
If people think deployment yamls are complicated, what til they see tekton.
Don’t get me wrong, I actually love tekton because I hate everything that is Jenkins and most other CICD tool integration with k8s. So tekton is just amazing in that regard.
But there is a huge learning curve for people who are used to old school tools like Jenkins and its pipelines .
Agreed. Where I work we have a core team that created a framework that manages the Kubernetes configuration for us. As a backend developer I basically just have to throw a config file into my service with some basic options like which port to listen on and where the docker container is, and then it gets deployed with everything setup for me (using helm I believe).
I really hope we move more towards these opinionated tools that can handle 90% of the use cases. Most people just want to host an app on a port, and it's a pain to have to develop that pipeline at every company that wants to adopt Kubernetes.
Agreed. Just like how I don't want to roll my own desktop environment on Linux, I will happily accept a good config rollef for me, even if opinionated.
I've always felt that Kubernetes isn't appropriate for most businesses to use directly, but rather it's a platform for simpler platforms--someone like Heroku would build on top of Kubernetes and expose a much simpler interface for their users so they don't have to think about SSL, DNS, logging, load balancing, auto-scaling, etc. Alternatively, maybe the problem is solved with "distributions" of Kubernetes analogous to Linux distros--preconfigured Kubernetes installations so you don't have to figure out how to configure your own Kubernetes service (arguably this is also what cloud providers give us with their managed k8s offerings). I'm curious to hear what others think.
100% agree. Kubernetes alone is just a """framework""" to describe your infrastructure, definitely not an "end-user product".
That is why in many businesses there is an OPS team managing the Kubernetes and providing tools like Cert-Manager, Istio, ... and the rest of the company who just use what the OPS team made.
Right now, everyone is building its own distro, proving IMHO the need for it.
Lots of PaaS offerings have been doing this for a long time, RedHat OpenShift being the original one; they contributed a lot of 1.0 k8s code and provided a lot of key early architectural feedback by implementing a PaaS using k8s. Another early one was Deis, who were at the first KubeCon.
All this is to say that you're at very least certainly half-correct, in that k8s is a very flexible tool that can be used to build a very simple, elegant, and ergonomic PaaS.
I'm not sure I agree that it's inappropriate for most businesses though, unless you think that only a PaaS like Heroku is appropriate for most businesses; the analogy I'd suggest is "Heroku vs. running your own VMs" circa 2010. Heroku is great for getting started, and lets you move fast by abstracting away a bunch of infra. But it's also restrictive; you can't pick and choose your components freely. As you grow past a certain point you'll almost certainly need the flexibility (or just cost-effectiveness) that you get from running your own infrastructure.
K8s is an improvement here because you can run a managed cluster on something like GKE, which takes away most of the operational toil, while still giving you a lot of flexibility on what components / pieces to include. The k8s domain API does a great job of abstracting away true infra concerns like volumes, compute, scheduling, load-balancing, etc, while making it really easy to package, use, and iterate on the stuff that sits on top of that infrastructure.
I'd probably not encourage a seed-stage startup to use k8s unless you're very familiar with the tool; a PaaS like Heroku would likely be more appropriate. However at the point that you'd usually graduate to running your own VMs (wherever that is on your own company's developmental path), I'd say that using k8s is now a better choice.
I think I generally agree. Maybe saying "most businesses" is a poor word choice; I don't really have the breadth of experience to speak for the majority of the industry; however, I do strongly suspect that a lot of companies are employing an ops team to wrestle with Kubernetes when their needs could probably be met by a PaaS, and I think this will only get more true as the first generation of PaaS-on-k8s arrive on the market in mature form. I generally agree with this:
> However at the point that you'd usually graduate to running your own VMs (wherever that is on your own company's developmental path), I'd say that using k8s is now a better choice.
However, even then there are intermediate options between VMs and full-on k8s, such as AWS ECS/Fargate, a Kubernetes distro, or managed Kubernetes offering (e.g., GKE) which give me the flexibility to interact with k8s, but they come with sane, pre-configured (or easily configured) solutions for logging, monitoring, ingress, load balancing, upgrades, etc.
I'm still building so I currently don't. Kubernetes is just a tool I use to deploy my customers sites on, it's not the selling point.
I do however have plans of supporting enterprise by packaging a version of the app into an on-prem deployment system so you can just deploy your own kubernetes cluster and install the app and have your own enterprise version of Primcloud.
Thanks for the shout-out! We’re trying to be like an “open-Heroku” - back when I worked on OpenStack it was called “the open cloud” and no one knew what we were on about - open stack was enterprise as enterprise gets!
Our goal is to make a Heroku-ish platform for getting an app online - but one that doesn’t hold you over a barrel later on. You can even host from a spare home server :)
I'm interested in/have previously given up on a product where the control plane is managed for me, I can join the cluster with bare metal nodes, and then it's just Kubernetes.
Yep - we expose the Kubernetes API (and add mutating controllers to help automate some of its actions).
Right now we don't do "hosted control plane", rather - you can either use our hosted clusters where you have namespace-level access (so you still get to use Kubernetes, just not all of the resources, for example, no `Nodes`), or you can attach your own cluster to our UI (with the advantage that we can setup ingress and forward traffic to you, which is perfect for a home-hosting setup).
Hosted control plane is something im watching closely though - I'd really love to offer that as a service, but since we're small, we're trying to focus hard on a core offering. Will be considered in the future though!
kubesail has really nice UI for an hosted option. Another alternative I found is https://kalm.dev/, which is open sourced and geared towards configuring a heroku on top of a k8s cluster.
Complexity always exists somewhere. You can't remove it, only try it make it easier to deal with.
What Kubernetes allows you to do is very complex so you need a capable system to express it all. YAML isn't always the best but it works fine for most and tools like these are very helpful.
Do you use an IDE? Does that mean the language and framework is too complex? No, it's just a tool to help you get things done. More tools aren't a bad thing.
>Complexity always exists somewhere. You can't remove it, only try it make it easier to deal with.
This is simply not true. Most complexity in today's systems is completely avoidable. Most developers are just mentally stuck and don't even try. "Managing" complexity is a great way to achieve job security without becoming good at anything specific. Instead of learning how to design system people are learning how to write config files.
That's a different argument. Not needing it doesn't mean it doesn’t exist.
If you don't need Kubernetes then don't use it. But if you do then you can't make it any simpler than the complexity needed to deliver the functionality.
Built in deployments, process health, logging, load balancing, security, high availability, config and secret management, volumes and persistence, and much more in exchange for some YAML files is a pretty good encapsulation of complexity though.
Agree with this, and would say that often a good way to deal with complexity is encapsulation. I imagine there are some developers that say "why wouldn't you just write it in assembly" Because if you are used to that and write small low level binaries it might work well. But trying to write a larger app it becomes more difficult to reason about.
Most of my colleagues find the yaml interface to be a lost cause. I tend to prefer to use the gRPC API as it's effectively the same declarative operations but you get to control it in software, which just feels right. YAML is just a crutch for specifying the API calls the client needs to make.
Ah right, guess I was thinking of the protobuf over HTTP API. In any case, I find using the protobuf types in your language of choice pretty ergonomic compared to YAML.
This is one thing that mesos/aurora got right (the "turing-complete-by-design" config files, written in Python). Templating yaml is like templating html. It works but absolutely sucks at scale if your job is to maintain it.
It's good to know that such an interface could conceivably be built for Kubernetes, using the HTTP API. Even better that it supports any programming language (since it's a network interface); I'd see that as a step-up from aurora configs, in fact.
At some point, I imagine this will become the default way that Smart Organizations build around k8s, once k8s's core API is super-stable and people develop sound frameworks in $POPULAR_LANGUAGE for declaring your configuration.
Then, we can then banish yaml to the same fate as json and xml. One can dream, at least. :)
I don't have an example offhand, but mentioned skycfg in the neighbor comment, which may be of some interest to you. I've used
https://github.com/ericchiang/k8s in the past with great success.
> Was Apache, Asterisk, or loading and hardening a Linux host on bare metal easier?
Yes, and by far. Adding a layer on top of all the traditional Linux daemons, tools and libraries does not decrease the total complexity - quite the contrary.
When you have a bug in an application that is related to something in on another layer you have to walk through the whole stack.
Examples: A bug in a network card impacting only large UDP packets. A race condition of file access triggered by NFS or a storage device driver. A vulnerability based on a timing attack due to CPU caches.
But wouldn’t the point be that you don’t care about hardware level problems anymore? When I find a node with issues, I can just delete it from the pool and get a fresh one back. The bad network card? That’s for Google/Amazon/DigitalOcean to deal with.
I find the bog-standard Prometheus chart provides me a pretty incredible level of monitoring out of the box, usually it’s pretty easy to pick the bad one out of a graph.
Running your own VMs without something like k8s? Yeah this setup I can deploy and have working in an hour is gonna take you a week to set up properly. Standardization is valuable. Abstraction is valuable.
> But wouldn’t the point be that you don’t care about hardware level problems anymore?
No. Read my post again: I did not wrote about hardware issues.
Most work around optimization, reliability or security require digging through the whole stack sometimes down to the kernel.
> When I find a node with issues, I can just delete it from the pool and get a fresh one back.
However, a lot of k8s deployments are on-premise, where you have to debug your own hardware.
> The bad network card? That’s for Google/Amazon/DigitalOcean to deal with.
First you have to pinpoint the root cause of that glitch affecting all the containers running on VMs using the same bad drivers. Often it could be the same in 50% of your fleet.
> is gonna take you a week to set up properly.
Most certainly not. I've been deploying large production fleets in minutes since 2005.
There are plenty of online tools to generate Nginx or Vagrant config files.
I don't really get the love for hating on K8 complexity on here - nobody says it's the perfect tool for every use case... but when you do need it, it's amazing, and has many advantages over the current trend for serverless IMO.
I find K8s to be the perfect tool for enterprises.
It provides the promise of portability, so the business feels less locked in to a vendor, while still being dense and impenetrable enough to them that IT won’t be done out of a job.
Simpler, more efficient alternatives are seen as more expensive, because IT hides the true cost of just how much of their salaries is spent of non-productive pottering around with yaml.
These systems and products also threaten to automate IT out of a job and don’t necessarily count much for their resumes. If you’re not a programmer what interest do you have in dumping all of your intricate, fragile environment for a PaaS? Not much
I always commit an example k8s YAML to any repo with k8s in it that describes how to get 2048 running.
Honestly, I get the pieces and I know where to look to get what so I'm not too bothered. The problem with wrapper tools is that sometimes I can't get at the insides and then I have to learn the wrapper tool. So I'm going to just stick to raw Kube until one of the wrappers wins out.
Isn't that the case with any tool or service of some complexity? I'd probably have to look at docs or examples for most config files i write.
There's trade-off between a tool being too specific to a usecase vs being too generic with the associated "boilerplate". But I don't know how much simpler Kubernetes could be made while still covering the intended scope?
Unfortunately, JSON is much more annoying to edit.
Best way is to stop using both, and generate the objects from higher level language, at least something like Jsonnet (which is really just a step up, so better not stop there but I will take what I can)
I'm on the fence about this, something like Dahl seems good, but I am not a fan of using fully-fledged higher level languages (the Pulumi approach) as I've been burnt before with devs writing bad code and generating configs in the most confusing, convoluted way possible.
It's a never-ending cycle. See: Front-end development. Until yet another abstraction emerges, anyone using Kubernetes for mission-critical applications should be using automated testing to prevent awful outcomes from a single botched line of YAML. There are a bunch of solutions emerging in this space, like Datree[0], made for the sole purpose of preventing Kubernetes misconfigurations.
"Simple" solutions to "complex" problems, by necessity, only work for certain subsets of the problems.
Modern distributed systems have tons of largely essential complexity.
The only way to reduce it, is to reduce the expressive power of the solution, which might be great for the cases where you fit into the mold, but then fails spectacularly (either with excess resource consumption, or by not working at all) when you push against the boundaries.
A lot of other "bleeding edge" technologies that blow up and become difficult to use end up with new abstractions, and then abstractions for the abstractions, and the original problem the tool was meant to solve is now not the focus any longer.
I don't think better abstraction is going to help the underlying problem: there are a lot of questions to answer, and you don't know the answer.
Without going into the complexity of Deployments, consider the lowly Pod. What configuration does your app need?
What is the name of the container that contains it? How much memory does it use? How much CPU does it need? What ports does it listen on? What HTTP endpoint handles the health check? Does that endpoint test liveness or readiness? What filesystems does it need? What setup needs to be done before the main container runs? Does it need any special resources like GPUs? The list goes on.
The problem here is that when you're writing a Pod spec, you're building a single-purpose computer from scratch. In the traditional UNIX world, people answered most of these questions for you. How much RAM can my app use? However much I plugged in. How much CPU can my app use? All of them. What ports does it listen on? Any of them from 1024-65535. What filesystems does it need? Whichever ones I setup in /etc/fstab.
I don't think it's a stretch to call UNIX's "yolo" approach problematic. It is great when you have one server running one app, but servers have gotten gigantic (with pricing to match) while applications have largely stayed the same size. This means you have to pack multiple apps onto one physical server, and to do that, there have to be rules. When you write a Kubernetes manifest, you are just answering every possible question upfront so that the entire system runs smoothly even if your individual component doesn't. It's the cost of having small apps on big computers.
The problem comes from applications that you didn't write, or don't fully understand. Before you can understand how the application behaves, you have to write a manifest. But you don't know the answers to the questions like how much CPU you're going to use, or what the worst case memory usage is, etc. This causes a lot of cognitive dissonance, because the entire file is you admitting to the computer that you have no idea how to configure it. No abstraction layer is going to fix that problem, except by hiding those uncomfortable details from you. (And you will always regret using the "yolo" defaults -- who hasn't tried to SSH into a broken server only to have Linux helpfully OOMKill sshd or your bash instance when you're just trying to kill your malfunctioning app.)
This is largely the fault of application developers. They aren't willing to commit to reasonable resource limits because they don't want to handle support requests that are related to underprovisioning. My experience is that applications that set limits pick them wrong. For example, GCP and DigitalOcean's managed Kubernetes offerings both install monitoring agents to support their dashboards; these apps ship with limits that are too low and any reasonable Prometheus installation will notice that they are being CPU throttled and warn you about it. Now you have to waste your day asking "is this a real problem?"
Many open-source apps go the other way and pick resource limits that truly encapsulate the worst case and require individual nodes that are many times larger than the entire cluster. Yes, it would be nice if I gave each pod 32 CPUs and 128GiB of RAM... but I don't want to pay $2000/month/replica thankyouverymuch. (I've been on the other side of that where resources didn't cost me real money and happily used terabytes of RAM as cache.)
Application-level configuration is also not in a great state. Everyone tries to sell you their curated defaults so they don't have to write any documentation beyond a "quick start". (I'm as guilty of that as anyone in fact!) The application will have some built-in defaults (so the developers writing the app can just "go run main.go" and get the config they need). Then someone comes along to make a Helm chart for you, and they change the defaults so that their local installation doesn't need any customization. This only causes problems because instead of an undocumented underlying application, now you have that AND an undocumented abstraction layer. You may find the answer to your question "how do I configure FooApp to bar?" but have no way of communicating that config through the Helm abstraction layer because the author of the Helm chart never thought anyone would do that.
This rant has gotten quite long so I'll wrap it up. No abstraction layer is ever going to make it so you don't need to answer difficult questions. The actual list of questions to answer is available through "kubectl explain pod.spec" and friends, however.
In my opinion, Kube's yaml blobs are comparable to binary programs. Complaining that an executable has its settings, paths, ports, memory limits, etc, hard coded inside of it is an obvious code smell.
Abstractions have a cost and a value. Programming languages are more expressive than writing out binary opcodes by hand, but don't really remove any features from the underlying machine. If your machine can do something, your C program can probably do something. So we use C. (It even ADDS the value that you can describe a procedure without knowledge of the underlying hardware. It may not be optimal on every machine, but at least you have a starting point for optimization!)
Helm, like any abstraction, has costs and values. For example, it's very valuable to be able to encode something like "my app consists of a frontend and a backend that run in the same Pod and must each use the same version of the code":
Now your frontend and backend containers can't get out of sync, and it saves someone from having to manually ensure that they are sync'd. There's no way to do it wrong! You supply {{ tag }} and it remembers the constraint that you required.
The problem with this abstraction is that there's no escape hatch. If you wanted to run different versions of foo-frontend and foo-backend, there is no way to say "but no, really, this time I'm violating the rules for a good reason". You've reduced the features available... the only way forward is to start over with nothing.
The result of this is that every individual Helm chart has to account for every possibility that the manifests could possibly encode, and invent their own programming language that is identical in expressiveness to the underlying manifests. And they do! Differently every time! For example, if I wanted to allow people to override the container images, I'd have to make my template look like:
Now it's possible, but not in a way that anyone could search for on the Internet. You will have to read the code or hope I wrote documentation for my ad-hoc Kubernetes extension.
I think we can all agree that didn't save anyone any time or effort.
This example conveniently flows into my other complaint with Helm. The "yaml files" that declare templates aren't actually valid YAML. You can't use something like `prettier` to autoformat them. You can't use the YAML language server to provide code advice as you type. You can't use `kubeval` to validate them. You are throwing all of that away to Build Your Own Thing. It is actually very insidious and for that reason I consider Helm to be more harmful than helpful. It isn't an abstraction, it's just a macro that might be good for one person the instant they happened to type something in.
The other problem is that Helm charts have no upgrade path. They are only designed for "please explode this project into my cluster, I promise to clean it up In The Future". It never gets cleaned up and brings a little piece of un-updated Windows 95 nostalgia right into your cluster.
Helm is actively harmful. And people love it, because it saves them a tiny bit of time one day at the cost of a lot of time in the future.
Templating within yaml is an absolute mess, I harped on ansible in the past for the same reason. However, dealing with that once is far, far better than dealing with thousands of hand-maintained kube yamls multiplied by your number of environments. Helm of course has its own pain points, but that doesn't subtract from the fact that it solves problems.
I see putting your example of two, unrelated, containers in the same pod as the same binary problem I mentioned earlier. I get your point, but it's a scenario one must wedge themselves into by making other poor choices. Why must the frontend and backend be the exact same version? The most obvious possible reason could be that the API used between them isn't versioned. Or maybe there's not even an API!
> You will have to read the code or hope I wrote documentation for my ad-hoc Kubernetes extension.
No, helm's approach to inserting variables into templates means this isn't the case. Every option and default appears in values.yaml and it's a one stop shop to see everything you can customize. The code example you wrote could be better written as:
Note that values.yaml is part of the template and values passed to helm's cli can override individual values in it.
I'm not sure what you mean about helm charts having no upgrade path. IME you can un-deploy, upgrade, and rollback helm deployments and it takes care of adding/removing kube resources that where also added or removed in the yaml for you. [1]
what's wrong with googling or copypasting ? toolset (helm template set) with basic blocks is added rather quickly, after that it could be as simple as running helm oneliner with needed variable values in the command line
Super cool to see these sorts of tools. They're great for learning the kube API.
For production-y things however, some meta-config language that allows deterministic templateing would be a huge improvement. It allows you to make sweeping/uniform infrastructure changes from a single library or tool change.
Kubecfg is a good example of the basics one could implement [0] although it's examples aren't as fully fledged and organized as they could be.
I can second using jsonnet and kubecfg for this. Not only you start out with some very useful abstraction from kube.libsonnet, but are also able to build your own abstractions easily, ones that are specific to you organization/codebase/release cycle. All from a declarative, lazily evaluated, pure language.
I've used Dhall in production, pushed it fairly hard and can say with utmost certainty it's be an absolute pleasure. We use it as our application configuration format and derive fairly intricate Kubernetes resources from our app config.
Although it is still in it's early days, it still is excellent to use and will only get better with additional tooling.
CDK uses existing fully featured programming languages that can preform side effecting actions thus lacking reproducibility and that are may not terminate. Dhall is a total language, meaning it will always terminate and a pure language, meaning that a function given an input will always yield the same output. That makes the output extremely predictable.
+1 to Pulumi. I can't imagine operating k8s without tools like Pulumi or Terraform now. And Pulumi addresses some of the most important shortcomings of Terraform so I really hope they gain more traction.
> we can then create functions where we would normally duplicate Yaml.
This is the way to go for sure. I've done similar by generating CloudFormation from Python (I wrote my own library because I felt Troposphere was not very friendly nor a significant improvement over YAML).
Typing turns out to be pretty useful when you're generating YAML. While my library was fully typed, Python's type checking left a lot to be desired--many stupidly common things still can't be expressed (JSON, kwarg callbacks, etc), getting mypy to find/accept type annotations for libraries is stupidly hard, and the IDE integrations are pretty awful at this point. TypeScript users would enjoy a real leg-up here since its type system isn't half baked.
> This is the way to go for sure. I've done similar by generating CloudFormation from Python (I wrote my own library because I felt Troposphere was not very friendly nor a significant improvement over YAML).
Yes and no.
Typing is a must, but a full-blown programming language is too powerful and all abstraction layers start to leak sooner rather than later. I always ended up with a "deployment" function that exposed almost all underlying functionality.
It depends. If you can reasonably trust your team to not to do I/O in the scripts, then you're fine. If you can't, then you should use something like Starlark (a subset of Python running in a constrained environment that precludes I/O, unbounded loops, etc); however, Starlark doesn't support types yet.
I've looked at Cue a few times (principally out of frustration with the lack of types in Starlark), but I don't really "get it". What I want is a typed embeddable scripting language--specifically the ability to expressively generate YAML, and I'm pretty sure that's not what Cue is. I'm open to the argument that Cue is better than what I want (a la "if Henry Ford asked people what they really want, they'd've said faster horses"), but the value proposition isn't clear to me from perusing the docs. Maybe you can correct me?
I have used a bunch of jsonnet and was in your position. I picked up https://jkcfg.github.io and have been very happy. You get TypeScript with (some) hermeticity guarantees.
Are you not worried about people writing arbitrary code to do stuff? I've been burnt before where devs used Turing complete languages (python in my case) to generate configs in probably the most convoluted and complicated manner possible. It was impossible to debug and understand, there were side-effects literally everywhere. It was everything you'd imagine from a normal bit of bad code, but it also happened to spin up hardware.
Pulumi looks good, but it confuses me. I thought it was like Ansible/Terraform, but I see a "pricing" page and it looks like there's a SaaS that goes with it, can someone shed some light?
If you print out the API docs for all the resources you might deal with for running apps at A4 (8.5 x 11) it comes in at over 1,000 pages. There is just so much and few really know it.
A UI like this is useful to so many. It makes the experience of creating these YAML files easier. Thanks for sharing it.
Paul, as an aside, we absolutely love how feature packed Octopus is nowadays. We have been using it since 2.0, and I don't think we will ever give it up. Thanks for making one of the best tools we use a daily basis!
I don’t know how to feel about Kubernetes configuration apparently being so complicated that you need a generator for it, instead of just having the docs and your IDE open in split screen like with Docker Compose.
That said, this still looks cool. I just hope we won’t need a Kubernetes configuration generator generator anytime soon.
Kubernetes manifests are no different from any code one would write. The first few times you need to accomplish something, you consult the docs. After a while, you know what you need. If you find yourself doing the same thing over and over again, automate it. I don't understand the hand-wringing over k8s YAML.
It's actually conceptually simpler than docker compose given what it does.
The bigger issue is that there's a lot of things you can do with it, and editing text YAML/JSON serialization of the objects is probably one of the least efficient ways of dealing with it, but it's also the "default" way people know.
It's much easier when your editor actually understands the object you're editing instead of at best helping you write correct YAML.
There are a number of such tools out there. Here's a short list. I'd be interested in any experiences people have had with them in largish production environments.
Hey, I'm the author of kube-score, and originally built the tool to support an organization using Kubernetes at a fairly large scale as measured in number of engineers, services, and Kubernetes clusters.
I'm obviously biased, but it's been hugely successful! kube-score is working very well out of the box, and there's only a handful of cases where the "ignore" annotation has been used to disable a check that's too strict for the particular use case.
Feel free to reach out if you have any questions or comments.
That looks super handy! It seems like it'd be possible to "port" to VSCode. Some other comments mention autocomplete in VSCode, but it'd be nifty to run something like this directly in the VSCode UI.
Ooooh nice! When I saw the title I was hoping this is what it would be. I'd dreamed of building something like this, but never had the time or the buy in.
Aside from making it easy to generate k8s manifests, this could also be a great learning tool. If you allowed this to generate multiple resources that are linked, it could be a great illustration of how different resources fit together.
All of the standard Kubernetes resource definitions have a schema [1] [2] which is already used to, for instance, generate Go code [3] and documentation [4].
Of course there is schema at some level, but it feels like it's not utilized. This UI seems like very good use of schema. Another would be if I open up an VSCode, there should be a way to autocomplete / or give error squiggles if I write something wrong, in realtime.
Right, but that's not as much of a Kubernetes issue as a VSCode issue. Everything's there for you to write something like this for your text editor, open source code doesn't grow on trees :).
My point being if k8s configuration was developed UI centrically, this problem would not exist in the first place. Because there would be official K8s GUI for configuration, one would not hunt and beg bits and pieces of docs every time editing a config file.
As a side note, usually the schema languages fail at some point, thus I referred to TypeScript interfaces, which are very flexible way to write validation.
There is already a schema support for JSON validation in VSCode, one can use it using `{ "$schema" : "https://example.com/url/toschema" }` but it uses JSON Schema format, which I think is not accurate enough in edge cases for UI generation.
If it was developed UI-centric, it would have failed much earlier to do anything useful.
The real power of Kubernetes is that it has very simple basic model that allows building complex constructs out of simple pieces, and in fact basic help (which is even used for UI in kubectl explain.
Do we need a better layer of abstraction, i.e. better adoption and tighter integration for something like kustomize? Have we fucked up completely with Kubernetes due to it being outrageously complicated for simple tasks? How we redesign this to be simpler? Is the complexity even a problem for the target audience?
I've no idea. I just know I'm a kubernetes admin and I can't write a deployment yaml without googling or copy/pasting.