In my opinion and in my experience, many actual programming languages are fine for defining config, and (due to you inevitably needing some control flow, and them having it built in) are often better than declarative languages. I for one have no problem with config in Python, JS, Bash, PHP, or Ruby. As long as it's in dedicated config files, and as long as it's kept as simple and as declarative as possible.
Every time I write software, in the beginning there are command line flags. Then someone goes, can I put them in a file. Then later, someone comes along and goes, it would be nice if that file could be dynamic in some way (i.e. let me insert stuff from environment variables). The next request is usually something like, could you let me have conditionals, so if SOME_VARIABLE is true one thing happens, and if false, another. Then someone comes along and goes, THIS IS DANGEROUS, probably NOT VERY SECURE, and your FORMAT SUCKS because you use some character I have to escape when templating, so could you use something modern and trendy, that totally doesn't suck like XML (or whatever format is in the engineering doghouse)?
What is funny is that a shell script does a pretty good job at giving you a nice, programmable way to invoke software. But, that is too complicated :-)
I believe what puts people off shell (myself included) are the various arcane caveats (eg. spaces after or before operators) and little language quirks. It does not seem intuitive and since there are better alternatives like Python, I won't bother to learn some 70s string-based language.
You can use it right now as a dev tool. If you use ShellCheck to statically check it, then running your script under Oil is complementary (and it also has some static checks): https://www.oilshell.org/why.html
Yeah I'm a big fan of Oil and hopefully in 10-20 years it'll be the standard language of shells. Also I did notice the spaces issues when attempting shell but it was only from reading your post sometime ago that I found out it was a deliberate language design choice.
It should be usable for "cloud config" files long before that! :) But yes changing existing code at the lower distro level has a lot more inertia.
The cloud is basically built on top of Linux distros, so that part is easier to change and is rapidly evolving.
You could say that the = issue is deliberate, but I'd say the core problem is that shell didn't start out as a programming language, or at least it was a very impoverished one without variables.
The original paper from the 70's on the Thompson shell shows that. Shell had "goto" but no variables! So name=value had to be grafted on later without breaking too many things. Words were already split by spaces, so I guess they just made
name=value
a "pseudo-word" that becomes an assignment, whereas
I highly recommend https://github.com/koalaman/shellcheck It's a wonderful piece of software to help and guide users writing shell scripts. Hook it up to your editor/IDE and all those arcane caveats will be things of the past.
"The next request is usually something like, could you let me have conditionals, so if SOME_VARIABLE is true one thing happens, and if false, another."
I remember one project that started out with simple XML config, then I added conditionals. when I was starting o work on loops and reusable variables I realized that I was writing a programming language in XML. So I started to write config in C# and compiled that dynamically instead.
I implemented a simulation framework once, where the core simulation process took in all parameters as command line options. A runner program would read a YAML file describing the parameters in arbitrarily complex ways, and invoke the sim process potentially thousands of times (doing parameter sweeps). The intent was for the underlying sim process to always be directly runnable for debugging just by copy/pasting the generated list of options, regardless of how complex the config file got. It was a year or so before somebody started passing in an intermediate config file to the sim process by command line argument...
This is actually one reason Ruby DSLs became so popular. You could take config from an XML file, translate it to the DSL, and have all the basic Ruby features (flow control included) available as well.
For example, there was an ANT library for JRuby that worked super well for using ANT constructs/libraries but in a sane language.
What is funny is that a shell script does a pretty good job at giving you a nice, programmable way to invoke software. But, that is too complicated :-)
Ha, I agree, although I also think shell is a bit impoverished. It works but there are valid reasons people don't use it.
Prediction: we'll see a lot more shell embedded in YAML in the coming years, with the same examples shown in the original article (Github Actions, didn't know about Helm Charts)
I've been round that loop a few times; if you use a full language for config then you either have to impose iron discipline or you sooner or later end up needing a configuration format for your configuration format.
That is to say the configuration eventually becomes a program in itself, with a few key values... which then get pulled out into a simple config file.
Depending on the use-case, I think you may be right. Especially if you can use a language that the team understands and has tooling for and you don't take in outside configuration.
Pulumi is an interesting tool in this direction. Rather than write in something like teraform, you just use your programming language of choice. Pulumi is just a library you use.
I think that depends on how complex the config can get, and how it is used.
For example package.json in NPM packages -> that feels like a good fit for JSON (although it would be even better if JSON had comments). On the other hand, terraform, or build languages like Make or Meson -> they are complex enough that it probably makes sense to have a standalone DSL.
I was facing the same decision recently on my project while designing a declarative DSL for web app development (kind of like web framework).
From simplest to most complex option:
- should I just let them define it all in JSON? There would be a lot of repetition at some point and it would become impractical, but it could be ok for the start.
- should I just implement JS library, that devs can use in JS to construct a config object that is then exported to JSON? That would be embedded DSL. Sounds flexible and easy to do, but it is also overly expressive and not "cool" (ok this is debatable).
- should I use something like Dhall? It is declarative and simple.
- should I come up with my own declarative, configuration-like DSL? It would probably end up similar to Dhall, but this means I can do whatever I want - I can make it as ergonomic and custom as I want to (which I guess is both good and bad :D!). It might also allow for nicer interop with Javascript and other languages.
In this case, we went for the last option, mostly because we felt the most important thing is ergonomics and interop, but well, I am still curious how would other directions play out. Plus at the end we didn't yet get to the point where language is more expressive than JSON (code example: https://github.com/wasp-lang/wasp/blob/master/examples/tutor...).
Maybe I am just missing a better design process, but it seems to me that with a language idea it is hard to say if it is good or not until you try using it.
But that does not allow you to manipulate it automatically (think about moving domains or something like this), nor does it help you to detect errors (which few tools do, but many actually should do).
So now you need the runtime for that other language in your system-wide configuration management solution. Plus all the other runtimes for languages someone decided to use as a configuration language.
Sounds alright to me, as long as you restrict to sane, reasonable languages. For example lua's "runtime" fits in a single .c file and is likely simpler than many xml or json parsers.
