Configuration Management is an anti-pattern. It's trying to constantly munge state to be back where you originally wanted it, rather than forcing it to remain one way and never deviate. All greenfield projects should be designed with version-controlled immutability in mind.
One example of how technology today is still kind of backwards is SQL databases. They have a crap operational model. Their schemas can mutate in odd, unversioned ways, and the data is pretty much a free-for-all, and both are lumped together in one weird mass. You can hack around it with flyway migrations, but this is a hack plain and simple. If it weren't for the convenience of the query language on simple datasets I'd never use them.
Similarly, trying to constantly "fix" the configuration of a system with a CM tool is like having a tea party at a mosh pit. Rather than having a tool designed to constantly correct a mutable system, you can instead build versioned, immutable artifacts, continuously deploy them, and roll them back immediately when tests fail. Even for entire OSes. Even on bare metal! But I guess because we don't have a lot of books on it yet, it doesn't seem to be the norm.
The industry hasn't really woken up to modern best practices. Take WordPress for example. Most WordPress installs are still these collections of manually curated random software and data in one big pot. There's no version control, there's no rollbacks, there's no testing... just back up everything all at once and restore it all if you break anything. And because of that, nobody updates anything, so security vulns abound. DevOps-oriented WordPress management has actually been around for at least 8 years, but almost nobody actually does it, because it's not immediately clear how, because the industry is stuck in two-decade-old paradigms.
Separation of DDL and DML in databases is pretty universal. The appropriate RBAC will stop schema modifications.
Have the information schema available allows automated deployments to introspect the database and generate the appropriate deltas between schemas and the migration of data.
SQL is a clumsy language, but don't throw the baby out with the bathwater.
Agreed. Any recommendations of the tools to use? NixOS/Guix come to mind, k8s (thought I have a mixed feelings about it). Anything else worth keeping in mind?
Kind of hard to cover it all, as it's sort of spread across many subjects, some of it is specific to a technology and some is more general. We really need a Wiki for it all. I tried to start one but it proved difficult to find other people who were interested. Here's some of the content: https://github.com/peterwwillis/devopsyoga-content/
This DevOps word soup will help point you towards good practices:
Infrastructure as Code. Immutable Infrastructure. Continuous Integration. Continuous Delivery. Test Driven Development. Blue/Green and Canary deployments. Cattle not Pets. Dead Cat syndrome. Not Invented Here syndrome. The Three Ways (Systems Thinking, Amplifying Feedback Loops, Continuous Learning and Experimentation). The Four Areas (Extend Delivery to Production, Extend Operation to Project, Embed Project(Dev) into Operations, Embed Production(Ops) into Project). CALMS. Andon cords. Autonomation. Value Stream Mapping. Shift Left. The Twelve-Factor App. Performance & Stress Testing. Quality Gates. Chaos Engineering. Idempotent & Atomic Deployments. Service Mesh. Distributed Tracing. Ephemeral credentials & secrets management.
Then again, a lot of best practices have no buzzword.
I don't know if "Simplified Engineering" has its own buzzword, but that's a best practice: use the least complex solution that meets your product's needs and also ensures other best practices. A Kubernetes cluster that isn't well managed may be much worse for security & reliability than running an app in a Docker container on a single pre-baked VPS. Static websites are more reliable than dynamic ones. And a static website served via S3 static site hosting is even more reliable [assuming it's not in us-east-1!].
Is the referenced issue something you really run across in etcd specifically? I mean, a more explicit error message would be great, but I wouldn't expect any database storage to not get corrupted if you start two databases with it at the same time.
Two instances could at least detect they are using the same conflicted file, and refrain from corrupting it. Likely the second one should terminate with an error.
NixOS is based on Nix, a purely functional package management system. by using Nix not only to build packages, but also things like configuration files.
By building entire system configurations from a Nix expression, NixOS ensures that such configurations don’t overwrite each other, can be rolled back, and so on.
I love nix and this mgmt thing looks cool but I really wish everyone would stop demanding I learn an esoteric new syntax to use their shiny useful new tools.
I simply cannot agree that their benefit is somehow inherently coupled to their DSL's syntax, and could not be equivalently modeled in an existing language's syntax so that users don't have to learn the Weird New Thing Of The Month.
Wrestling with the nix REPL is currently my bane, despite being somewhat in love with NixOS and running it on my desktop and in containers.
The first visible advantage of GUIX over NIX is the use of scheme language. But over here it is probably as esoteric as nix. It's funny to think that if they had settled on Javascript instead, they might have become instantly the next unicorn.
Which makes me wonder what happened to Guile's Javascript front end, and would it work with GUIX?
It's also funny to consider that JavaScript came to be because Netscape failed to implement Scheme in the browser. Imagine what the world would look like had they succeeded with that... :-)
Surely we would hate scheme as much as javascript, complain about how all Joe programmer really wanted was a mere way to have a familiar nominal object system and not that sophisticated call-cc thing, and complain about the lack of compile time type checks.
Whenever I see a new piece of technology, my first question is always why is this better than <insert standard tech>?
Now, I'm not super familiar with other configuration management systems, but I really have to dig through the docs/blog posts to figure out what the core differences are. Would be nice if the differences were more clearly delineated.
This is going to be a really shallow answer but all the existing solutions are horribly engineered. Even relative newcomers like SaltStack and Ansible are a horrible mess of inconsistent behavior and spaghetti code.
There's nothing special indicating that mgmt will be any different, and it seems to me like the project isn't very active, and I would say the era of complex configuration management is over (moving more towards immutable infra, containers and k8s) so it's not worth the effort.
If I were to try and give an answer to your question though: mgmt has a dedicated DSL (like Puppet, but unlike Salt and Ansible which is a mess of YAML + Jinja) but without the legacy code and the ability to parallelize execution of states.
I've worked with Puppet and Ansible, and now I'm a user of Saltstack, and don't often hit my toe on the problems you're asserting.
One person's legacy code (in the pejorative) may be another's tried, tested, robust core.
Similarly, the 'mess of YAML + Jinja' - might be seen by others as leveraging the power of both. (And in Salt, recipes can be almost anything, eg vanilla python, so long as they spit out something that parses as config.)
My gut feel is DSLs really have to offer something hugely compelling in order for me to want to abandon conventional and common config markup formats.
Seems like containers and kubernetes is the newer method rather than stateful and mutable cfm controlled boxes. Love Nix though in comparison to either approach. Nothing quite as amazing as your entire box effectively coded via nix to be setup a particular way. Makes me wish it were even more popular than it's already becoming.
One example of how technology today is still kind of backwards is SQL databases. They have a crap operational model. Their schemas can mutate in odd, unversioned ways, and the data is pretty much a free-for-all, and both are lumped together in one weird mass. You can hack around it with flyway migrations, but this is a hack plain and simple. If it weren't for the convenience of the query language on simple datasets I'd never use them.
Similarly, trying to constantly "fix" the configuration of a system with a CM tool is like having a tea party at a mosh pit. Rather than having a tool designed to constantly correct a mutable system, you can instead build versioned, immutable artifacts, continuously deploy them, and roll them back immediately when tests fail. Even for entire OSes. Even on bare metal! But I guess because we don't have a lot of books on it yet, it doesn't seem to be the norm.
The industry hasn't really woken up to modern best practices. Take WordPress for example. Most WordPress installs are still these collections of manually curated random software and data in one big pot. There's no version control, there's no rollbacks, there's no testing... just back up everything all at once and restore it all if you break anything. And because of that, nobody updates anything, so security vulns abound. DevOps-oriented WordPress management has actually been around for at least 8 years, but almost nobody actually does it, because it's not immediately clear how, because the industry is stuck in two-decade-old paradigms.