Configuration Management is an anti-pattern. It's trying to constantly munge state to be back where you originally wanted it, rather than forcing it to remain one way and never deviate. All greenfield projects should be designed with version-controlled immutability in mind.
One example of how technology today is still kind of backwards is SQL databases. They have a crap operational model. Their schemas can mutate in odd, unversioned ways, and the data is pretty much a free-for-all, and both are lumped together in one weird mass. You can hack around it with flyway migrations, but this is a hack plain and simple. If it weren't for the convenience of the query language on simple datasets I'd never use them.
Similarly, trying to constantly "fix" the configuration of a system with a CM tool is like having a tea party at a mosh pit. Rather than having a tool designed to constantly correct a mutable system, you can instead build versioned, immutable artifacts, continuously deploy them, and roll them back immediately when tests fail. Even for entire OSes. Even on bare metal! But I guess because we don't have a lot of books on it yet, it doesn't seem to be the norm.
The industry hasn't really woken up to modern best practices. Take WordPress for example. Most WordPress installs are still these collections of manually curated random software and data in one big pot. There's no version control, there's no rollbacks, there's no testing... just back up everything all at once and restore it all if you break anything. And because of that, nobody updates anything, so security vulns abound. DevOps-oriented WordPress management has actually been around for at least 8 years, but almost nobody actually does it, because it's not immediately clear how, because the industry is stuck in two-decade-old paradigms.
Separation of DDL and DML in databases is pretty universal. The appropriate RBAC will stop schema modifications.
Have the information schema available allows automated deployments to introspect the database and generate the appropriate deltas between schemas and the migration of data.
SQL is a clumsy language, but don't throw the baby out with the bathwater.
Agreed. Any recommendations of the tools to use? NixOS/Guix come to mind, k8s (thought I have a mixed feelings about it). Anything else worth keeping in mind?
Kind of hard to cover it all, as it's sort of spread across many subjects, some of it is specific to a technology and some is more general. We really need a Wiki for it all. I tried to start one but it proved difficult to find other people who were interested. Here's some of the content: https://github.com/peterwwillis/devopsyoga-content/
This DevOps word soup will help point you towards good practices:
Infrastructure as Code. Immutable Infrastructure. Continuous Integration. Continuous Delivery. Test Driven Development. Blue/Green and Canary deployments. Cattle not Pets. Dead Cat syndrome. Not Invented Here syndrome. The Three Ways (Systems Thinking, Amplifying Feedback Loops, Continuous Learning and Experimentation). The Four Areas (Extend Delivery to Production, Extend Operation to Project, Embed Project(Dev) into Operations, Embed Production(Ops) into Project). CALMS. Andon cords. Autonomation. Value Stream Mapping. Shift Left. The Twelve-Factor App. Performance & Stress Testing. Quality Gates. Chaos Engineering. Idempotent & Atomic Deployments. Service Mesh. Distributed Tracing. Ephemeral credentials & secrets management.
Then again, a lot of best practices have no buzzword.
I don't know if "Simplified Engineering" has its own buzzword, but that's a best practice: use the least complex solution that meets your product's needs and also ensures other best practices. A Kubernetes cluster that isn't well managed may be much worse for security & reliability than running an app in a Docker container on a single pre-baked VPS. Static websites are more reliable than dynamic ones. And a static website served via S3 static site hosting is even more reliable [assuming it's not in us-east-1!].
One example of how technology today is still kind of backwards is SQL databases. They have a crap operational model. Their schemas can mutate in odd, unversioned ways, and the data is pretty much a free-for-all, and both are lumped together in one weird mass. You can hack around it with flyway migrations, but this is a hack plain and simple. If it weren't for the convenience of the query language on simple datasets I'd never use them.
Similarly, trying to constantly "fix" the configuration of a system with a CM tool is like having a tea party at a mosh pit. Rather than having a tool designed to constantly correct a mutable system, you can instead build versioned, immutable artifacts, continuously deploy them, and roll them back immediately when tests fail. Even for entire OSes. Even on bare metal! But I guess because we don't have a lot of books on it yet, it doesn't seem to be the norm.
The industry hasn't really woken up to modern best practices. Take WordPress for example. Most WordPress installs are still these collections of manually curated random software and data in one big pot. There's no version control, there's no rollbacks, there's no testing... just back up everything all at once and restore it all if you break anything. And because of that, nobody updates anything, so security vulns abound. DevOps-oriented WordPress management has actually been around for at least 8 years, but almost nobody actually does it, because it's not immediately clear how, because the industry is stuck in two-decade-old paradigms.