The idea of declarative config is that empirically programmatic config was bad at scale.
If your config is the source of truth of what your infra should be, then you can use source control tools to roll back to a known good state, or to binary search for when a problem was introduced.
If you use programmatic config, then you can't find out the intended state of your system without executing a program. You can't grep through program executions in a meaningful way, especially at scale. So you can't do even simple things like search for a string.
Guaranteeing termination is helpful, but it doesn't solve the main problem that programmatic config puts a large complexity barrier between you and the ability to understand your infrastructure.
Tools like Helm give up a fair amount of this declarative benefit. And IMO that's one of the reasons why it's almost always a worse experience to use a helm chart than to just render the chart once and for all and forget Helm ever existed.
> The idea of declarative config is that empirically programmatic config was bad at scale.
Languages can be declarative or imperative. For instance, Pulumi and CDK are declarative.
> If you use programmatic config, then you can't find out the intended state of your system without executing a program. You can't grep through program executions in a meaningful way, especially at scale. So you can't do even simple things like search for a string.
I don’t understand — nothing stops a language from having an intermediate compilation step that shows the intended state and is searchable. Beyond that, programmatic config means you can add in hooks to make plans or runs more interrogatable.
> Guaranteeing termination is helpful, but it doesn't solve the main problem that programmatic config puts a large complexity barrier between you and the ability to understand your infrastructure.
It seems like this is untrue — having seen templated IaC that is hundreds of thousands of lines and cdk that defers that complexity to an abstraction that I have to understand once, I’d always take the latter.
Agreed that helm use is a giant mistake and liability.
I think codegen/compilation is a middle ground here. A higher level language like starlark can be compiled down to a set of instructions that provide the described guarantees.
This is how Pants (build system) works. You have declarative Starlark which supports basic programming semantics and this generates a state the engine reads and tries to produce.
I've been meaning to dive into jsonnet for a while but it'd be good to have a higher level representation that didn't rely on sophisticated templating and substitution engines like current k8s.
Compare k8s to Terraform where you have modules, composability, variables. These can be achieved in k8s but you need to layer more tooling on (kustomize, helm, etc). There could be a richer config system than "shove it in YAML"
Things like explicit ordering and dependencies are hard to represent in pure yaml since they're ",just text fields" without additional tools
If you restrict your language to pure functions only, then it is quite possible to have a system be both declarative and reproducible while having more expressivity than yaml.
I've tried out Pkl which is similar in spirit, and I think it's a real solution for k8s manifests. The only thing holding it back is industry adoption imo. It's leagues better than Helm, and mostly better than Kustomize.
See also: KCL, which is very similar and might _actually_ be "the winner". Time will tell.
I don't expect a winner personally, rather that there will be dozens of alternatives always. Like build systems, deployments are quite bespoke to organizations and legacy has a way of sticking around for a long time
Having used CUE, mainly outside of Kubernetes, I cannot see myself switching to KCL. I really like having a configuration language that isn't so tied to a specific system and which I can use with the Go SDK
If I was in charge of our infra automation I would have done this. We opted for jsonnet instead which is an absolute nightmare, or at least the way we've set it up is.
My take on this is that the issue is not declarative infrastructure resources, but a tendency to over-complicate the infrastructure.
For example: You have a problem that is suitable for some message queue -> Apache Kafka. Now you have 7 new problems and the complexity warrants perhaps 3 other services, and on, and on.
Introducing complexity is always something that needs to be introduced carefully. It makes things harder if you introduce it to early, but everything will break in a big bang if you introduce it too late.
Nowadays you can also start with a light weight MQ like Rabbit MQ and decouple your service just into a hand full components. This will set you up for scalability without introducing massive overheads.
In end it is also always a knowledge game. How experienced are you or how much time are you willing to invest into learning and understanding a technology.
I have been developing my own package manager, and my core idea is that proper programming languages are the proper level for describing packages.
Programs take inputs and can output arbitrary data such as resources. However they can do so with type safety, and everything else a programming ecosystem can achieve.
For asset distribution it uses wasm, and that's it!
Because people write code that is hard to understand. Configuration doesn't need all that. What it needs is to be provably correct and easy for someone to make predictable changes under high pressure (when prod is down). The non-terminating thing is one of the features of a turing incomplete language, not the goal. You don't want inheritance either, because it becomes hard to know where and when a value gets set (which is what helm overlay via multiple -f uses effectively is)
You speak like turning incomplete languages cannot have the control structures, tooling, and ecosystems we enjoy elsewhere, which would be the wrong assessment. I recommend you take a look at CUE to see how this can be true
The OpenAPI specs are probably better than the Go language types for k8s. They have more of the validation information and you can get at the CRDs / versions actually running in the cluster.
I am not saying that Turing incomplete languages don’t or can’t be a good fit for this task.
However there’s no reason we should rule out general purpose languages.
We have a lot of configuration based IaC and configuration tooling a la jsonnette and cue and yet these are riddled with their own problems and DX issues.
Anyways we don’t need to see eye to to eye on this but I respect your position.
> However there’s no reason we should rule out general purpose languages.
We’ve learned the hard way general purpose languages are poor for configuration at scale. I know first hand having worked on some of the larger prod infrastructures out there.
At scale, the best SRE’s out there still have trouble reasoning about the system and end up pushing bad config that takes down prod.
Languages like CUE really are different and better. CUE in particular hits the right balance for configuration of millions of lines of k8s yaml.
I actually really like CUE. I use it kind of extensively as a YAML replacement where I can, and at my work we've done our best to integrate cue with our charts to validate the values used to invoke our charts and to unify the value space.
However there's something about a full blown general purpose language that is so much more flexible.
I don't think that the fact that people can and do write bad programs disqualifies general purpose languages from being great tools to build packages.
I am sure there is just as equally bad CUE, Jsonette, PKL, etc out there.
Other than CDK8s I don't know of other tools that have tried in this space to use general purpose languages to define their packages, and I think CDk8s uses are generally happy. Much more so than helm users at least.
I am not sure I can agree with this statement
> We’ve learned the hard way general purpose languages are poor for configuration at scale
I think we've just assumed this, or seen a pulumi project we didn't like working in.
I believe and hope there will be plenty of room to experiment and innovate in this space!
The difference is in shared understanding. With tools like CUE or Starlark, you can learn one system and everyone can reason across each other's work. With imperative languages, every instance is a snowflake and creates significant mental overhead. It's the flexibility that is actually the problem for configuration. I get there is/was a trend towards general purpose languages in DevOps, but I think we are post peak on the adventure.
I think proper programming language support is the way to go.
Ideally a static type system that isn't turing complete and guaranteed to terminate. So something like starlark with types.