Hey, original author here. Thanks for sharing this and making it rise to the front page! :) By the way, the title probably deserves a (2020) and it would be nice if "test" wasn't capitalized, because it actually refers to the command.
[[ is not really a builtin, it's fundamentally syntactical (but presumably uses a mostly-inaccessible builtin internally). Fun fact, `]]` is also a reserved word despite never being allowed in a context where reserved words matter.
In some non-bash shells, the `function` keyword is needed to declare certain types of function.
For make `$(shell)`, if you're building a lot of targets the performance difference can be measurable. Still, it loses in the nop case, so you should actually usually do `include` to trigger re-making.
GNU is completely right to ignore POSIX, since POSIX is not useful for solving most real problems.
A lot of the GNUisms described in https://jmmv.dev/2021/08/useless-use-of-gnu.html are very useful in interactive use. Search the current directory without an explcit . - useful. Append an option to a command you just typed - heck yeah, always annoyed when commands don't support that.
For scripts though, sticking to POSIX sh often makes sense, yeah. You should at least be aware if you use of Bash-isms.
This article complains that using the extended GNU features (--ignore-case, set -o pipefail etc) makes scripts less portable. Fair enough.
What it doesn't explain is why a Linux user should much care about portability. OpenBSD and FreeBSD are alive and well, but the number of users seems so small that they aren't a particular concern. Maybe you could argue that we "should" consider these OSes out of a sense of fairness, but where does that stop? Do I also need to consider something obscure like vxWorks?
BusyBox (Alpine) is more interesting, but the changes there are so significant that a port will almost always be needed anyway.
Are there other compelling reasons to care about the non-GNU ecosystem?
I find the obsession with shell-script portability in contexts where it doesn't matter to be bizarre, but this particular argument is amusing:
> the number of users seems so small that they aren't a particular concern. Maybe you could argue that we "should" consider these OSes out of a sense of fairness, but where does that stop?
This is the same argument that many companies and products have made over the years (and still do at times) for ignoring Linux. To have a Linux user use that same argument against OSes with even smaller userbases is kind of amusing to see.
What I'm trying to say is that open standards (and the portability that comes with them) is not something that just happens on its own. It takes active maintenance, and part of that maintenance is opting to adhere to the standard even when it would be more convenient to use extensions available in the most popular systems.
Will you personally suffer from liberally using Bashisms? Not in the first order. But if we encourage that sort of thinking as a rule, the standards become meaningless. I believe that would be a net negative change for the world, but there are many intelligent people who would disagree.
I don't disagree with you in the context of public scripts, but the vast majority of what I do with bash is local-only (where local is ${DayJob} and my own infrastructure/projects) so the fact that extensions to the standard make my work in those areas quicker to string together and easier to maintain afterwards means I'm happy to lock myself into bash.
From my PoV it is generally portable anyway: almost everywhere I use a shell of that nature bash is present. The exceptions to this are things that need to run in small environments, like anything which may end up in initrd (where busybox us generally providing shell/script support).
Though I do wish people would make sure they specify bash in the #!/bin/bash in scripts unless trying to stick to the standard, as using #!/bin/sh causes problems when extensions are used and something lighter than bash is the default script runner (i.e. dash in Debian, and again busybox commonly in small environments). Only use #!/bin/sh if you lnow you are making an effort to be compliant.
> I don't disagree with you in the context of public [things], but [...] extensions to the standard make my work in [private] areas [...] happy to lock myself into [a non-standard].
This is the best argument I know in favour of not standardising, and the danger is not the direct consequences, but rather than our expertise in the standard atrophies when we do most of our work outside of it, which means our public work suffers too.
Also sometimes the private goes public and then who's gonna fund redesigning it in the standard? That rarely happens.
> From my PoV it is generally portable anyway: almost everywhere I use a shell of that nature bash is present.
A key difference is that bash is not claiming to be standard where it is not. It implements the full standard as well as its own extensions. It does not do standard things wrong¹ in order for its extensions to work².
I see posix-sh/bash more like javascript/typescript than good-browsers/internet-explorer. Which makes me wonder: is there a gnu/bash->posix/sh converter out there…
--
[1] well, there are a few oddities, like == behaviour in test which can cause portability issues because the standard doesn't specify that and people might accidentally use it when trying to be as portable as possible, but this also affects sh so is a general environment issue not specifically a bash related one. Also that doesn't break standard portable behaviour.
[2] or in the case of IE, doing standard things wrong, and its own extension wrong between versions too!
> Is your comment best read in Internet Explorer 6?
I actually feel triggered.
That was such a nightmare period in my web-dev career (pretty much the entire 2000's), it pretty much singlehandedly pushed me to find backend work as much as humanly possible
You will notice that the parent article mentions "dash."
The dash shell is small and fast, but it does not allow any bash/korn language extensions beyond what was recorded in the POSIX.2 standard in the early 1990s.
Linux users should care because the Debian/Ubuntu family use dash as the system shell, so this problem is very real as many have learned.
`-o pipefail` isn’t a GNU extension or even a difference in user space programs, but specifically a bash option. The portability is between shells, not operating systems. It will work on bash, regardless of the underlying OS, but may or may not work on csh, ksh, fish, etc.
Mac's command-line tools are the BSD versions. I always get confused about the command-line arguments for `date` because they differ so much betwen platforms.
if [ a = b ] || grep -q ^hello$ /usr/share/dict/words; then
echo "test failed and grep succeeded"
fi
> “You pick whether to be amused or horrified. I don’t know how exactly my coworker reacted when I hinted at this during a recent code review I did for them.”
Tangential, but your blog software seems to have mangled the headings; it's serving eg:
make $(shell …) expansion
instead of
make $(shell ...) expansion
(note that (as is correctly written in the body) that's three pediods, not one elipsis, so mldr wouldn't be correct even on it's own, meaning there's two probably-unrelated bugs affecting it).
Hmm… I’ll have to check later but right now I’m seeing the right thing in iOS. The blog is built by Hugo, so it’s all static files. But maybe something changed with the latest update. Thanks.
The rabbit hole of why "mldr" (em length leader) is a thing vs. "hellip" (horizontal ellipsis) was pretty disappointing. I only decoded "mldr" and didn't find much in the way of an explanation for why both entities exist.
I assume there's some semantic reason in which each is appropriate. I don't know what it is and am curious though!
> [ a = b ] && echo "Oops!" || echo "Expected; phew!"
Not to be taken as a general rule though. I might be mistaken but I think that bash would parse the line as:
([ a = b ] && echo "Oops!") || echo "Expected; phew!"
so if the command sequence after `&&` fails, then the code sequence after `||` is executed anyway:
illo@joe:~ $ [ "a" == "a" ] && >/dev/full echo "strings match" || echo "strings don't match"
-bash: echo: write error: No space left on device
strings don't match
illo@joe:~ $
This is different from the semantics of the `if` block:
illo@joe:~ $ if [ "a" == "a" ]; then >/dev/full echo "strings match"; else echo "strings don't match"; fi
-bash: echo: write error: No space left on device
illo@joe:~ $
No it won't. set -e is implicit disabled for the first command with && and ||. Same for a command after if/while/until and after !.
It should only matter if you implicit return immediately after.
Arithmetic expansion is a feature of all POSIX compliant shells, which furthermore advises against using `expr`. The latter is only needed for non-compliant shells like some legacy implementations of the Bourne shell.
What's the basis for that assumption? I'm struggling to see a reason to not use the shell for arithmetic, regardless of `/usr/bin/[`, builtin `[` or `[[`.
I stopped using [ a few years ago because ‘test’ reinforces the idea that this is just a command like any other, not syntax. Also, “man test” is much more pleasant that sifting through “man bash”.
I think GP's point was that `[` feels like syntax, but - importantly - isn't.
Yes, `[` is a command, and has a man page and everything, but in a script it doesn't look like a command. It looks like something "special" is going on.
Whereas using `test` emphasises the point that you're just running another program. That all `if` does is check the output status of whatever program is being run, whether that's `test`/`[`, or `grep`, or anything else at all.
(Personally, I don't think that emphasis is necessary. But I've been using shell scripts long enough that I understand these nuances fairly well without having to think about them much any more. So I think that GP's point is a reasonable perspective to have, and worth considering.)
It's not working for me in a non-interactive Bash script on Bash 4.4.
$_ is nto documented in the man page but is in the Info manual. It is not only set to the last argument of a command but also to the script name on script startup.
$ bash --version
GNU bash, version 4.4.20(1)-release (i686-pc-linux-gnu)
[ ... ]
Contents of script:
$ cat underscore.sh
#!/bin/sh
# ^^ mistake here, should be /bin/bash
test -e foo && echo $_ exists
echo 'value of $_' = $_
Test:
$ touch foo
$ ./underscore.sh
./underscore.sh exists
value of $_ = ./underscore.sh
I'm seeing nothing but the behavior of $_ being set to the script, and not affected.
But at the interactive prompt:
$ test -e foo && echo $_ exists
foo exists
This doesn't look like something I can rely on in scripts.
In the first place, I code in POSIX, except in the rare situation of making something that is strictly geared toward Bash.
Magic global variables subject to hidden side effects are garbage; I already feel dirty enough when I have to use $?.
This piece of crap also breaks under the DEBUG trap:
$ debug() {
> :
> }
$ trap debug DEBUG
$ test -e foo && echo $_ exists
debug exists
$ test -e bar && echo $_ exists
!1!
(That !1! is how non-zero exit status is printed in my setup.)
Sorry, I'm not going back to a 1978 way of writing shell tests, in order to use some broken magic variable.
I'm with you, the [ (and [[ bashism) introduces lots of confusions about what is really happening, I could never manage any real confidence.
That said, [[ being guaranteed to be built-in certainly had its purpose at ages where shell script performance had any kind of relevance, and that was no so long ago.
[[ has more to do with trying to build an intuitive shell scripting environment than performance. [[ makes conditionals behave much more like you'd expect from other programming languages. I think it's a great idea, but then again, if I don't have to care for POSIX portability, I'd rather use something that's not a shell language for scripting.
While they were at it, just to be consistent, they should have added syntax for easy-to-use non-fucked-up less-arbitrarily-punctuated versions of control structures, too:
ifif [[ x == 1 ]] thenthen
echo "x is one"
elseelse
echo "x is not one"
fifi
forfor x in 1 2 3 dodo
echo "x is $x"
donedone
whilewhile [[ x == 1 ]] dodo
echo "x is one"
x=$((x + 1))
donedone
untiluntil [[ x != 1 ]] dodo
echo "x is not one"
donedone
The whole [[ ]] $(( )) ifif fifi dodo thing just seems like they're just doubling down instead of admitting they made a mistake.
And if they really wanted [[ ]] to seem like syntax instead of a shell command, they could at least allow it to be used without spaces on each side of it like $(( )) or parens or brackets in any other language. And every other language lets you use as many redundant parens as you like for clarity, without running twice as many commands or producing a syntax error or weird unexpected behavior. But I don't think clarity was ever a design goal with Unix shell scripting languages.
> arbitrarily-punctuated versions of control structures
Ahem:
if [ x = 1 ]
then echo "x is one"
else echo "x is not one"
echo "namely, it's $x"
fi
`if [ ] ; then` should only ever be used in one-liners, where there is not a newline after `then`; I'm not sure how that ended up being taught as a way to write multi-line commands (I'd tenatively blame Pascal, but that's probably unfair).
I’ll probably just use test after reading this, I write bash scripts infrequently enough that if statements always get me (whitespace). Now that i see the why it’s super obvious, and using test helps communicate that it’s just args
The biggest footgun in `[` and `test` is the single argument behavior. For example, you might attempt to check if a variable is nonempty like so:
[ -n $FOO ]
but if FOO is unset, it expands to nothing (as opposed to the empty string), so this is equvalent to:
[ -n ]
and POSIX requires that the one-argument form of `[` succeed if that argument (here, "-n") is non-empty. So this will falsely report that $FOO is non-empty.
I think your last sentence needs to go first. Quote your variables! There's no actual footgun in the specification of the test builtin -- the footgun is shell itself. The behaviour that you mention makes sense because
[ "$FOO" ]
is always the non-empty check regardless what it contains (could be "-n").
The form ${var-} form is useful for safely evaluating a variable that might be unset, when "set -u" mode is in effect, and for whatever reason we cannot just fix the script so that the variable is set.
> and for whatever reason we cannot just fix the script so that the variable is set
I use this pattern all the time for variables that should be overridable by whoever is calling the script, ie. `FORCE="${FORCE-no}"`, overridable with `FORCE=yes foo.sh`. I'm not sure of any other way to do this.
(Yes, using getopt or other option parsing is better, but for my own scripts, env vars are just such a simpler way to pass options, and after 20 years of using bash/unix/linux I still can't write a getopt stanza from memory.)
while getopts :hab:c o; do
case $o in
a) opt_a=x ;;
b) opt_b="$OPTARG" ;;
c) opt_c=x ;;
h) usage 0 ;;
?) usage 1 $o "$OPTARG" ;;
esac
done
shift $((OPTIND - 1))
There was definitely a wink component, but at the same time, I meant it.
Half the code is just case / esac syntax, which you can (i do) find plenty use for outside of getopts. getopts itself is… manageable. In the end, the code is such a strong pattern, it's basically a snippet. The lack of variation is what makes it kinda easy to remember. Or you could put it in your notes or a gist and copy/paste.
> That's super difficult to do from memory, at least for me. Maybe I'm not as good at memorizing things as you are.
Nah, my memory is shot. It's a matter of practice, and not being shy with man pages. Use it often enough, it's in memory; after a long hiatus, the details are a "/^\s*getopts \[" away (assuming your man pager is less).
I find it's much better to have a copy-pasted piece of code which turns all command line options into variables in a certain namespace (provided the variables are defined beforehand)
However, the variable opt_uiuez doesn't exist and so it bails:
script: no such option: --uiuez
With such a piece of code, all you do is define the options you support via assignments like "opt_foo=". You can give them default values this way. Include that piece of boiler-plate code. Done.
To add a new option, just define the variable. Done.
Check its value wherever needed, and possibly the _given, if the code needs to know whether it's working with the default value, or an explicitly given value. That's it.
"set -x" is a life saver for debugging. Also "set -euo pipefail" so a script just exits on errors or unexpected behavior. If done well you can really reign in squirrely behavior.
Bash has some odd behaviors and footguns but it can also be surprisingly reliable and versatile. Ive caught some seriously messed up stuff that I couldn't figure out otherwise, using set -x. Also shellcheck will forcefully hammer good practice into your head and catch a ton of hangups that are hard to know before making the mistakes.
also setting backup variables is a good idea like:
pictures="${pics_dir:-$HOME/Pictures}"
chubot has written an interesting document¹ exploring more of the nuance with test/[/[[, and many of the other entries in that blog have intriguing explanations of the oddities of our shells(a random example²).
"bash only" typically refers to "bashisms", that is, bash features not present in the plain Bourne shell (or Bourne-compatible interpreters such as dash). The fact that other shells (such as zsh) may include those features ... is beside the point of writing universally compatible shell scripts.
Confirming my facts for this comment, #TIL that "dash" is the "Debian Alquist Shell", that is, Debian's "ash" shell:
While zsh, bash, mksh, ksh93, probably others have it, sure. But many don't -- and not totally irrelevant ones either. Debian's default, dash, for example, does not support `[[`.
It's usually pretty trivial to avoid them, especially if you're willing to call other mandated commands like awk, etc. But often, with a bit of creative thinking, most non-standard features can be replicated with some combination of `set`, separate functions and/or subshells.
Shell scripts, in general, have dozens of footguns, are pretty much impossible to statically analyze, difficult to make truly robust, and many of the shells themselves -- e.g., bash -- have huge, borderline inauditable codebases.
I can think of a dozen reasons not to write shell scripts. Yet still, there is incredible value in the fact that some form of POSIX-compliant/compliant-enough shell can usually be found on most systems.
All of that value goes out the window, though, the moment you start relying on non-standard features.
If you're writing an installer script or whatever that's going to be run on $many computers that you don't control: sure, it's probably best to go to the extra effort to stick with POSIX sh.
But that's not most scripts. Most scripts are things that you run on computers you control. I just write things as zsh scripts because it's so much easier than bash (never mind POSIX sh). That's fine, because I can just install zsh. And actually, this often makes scripts more portable because you need to rely on external utilities a lot less.
Don’t worship at the altar of portability. There is real cost to portability, as it forces you to cater to broken implementations and to not use features that may be very useful. Also, it can be difficult to ensure compatibility with systems that you don’t test on, so true portability requires having different systems to test on.
Sometimes all those costs are necessary for the task at hand. For example, the whole point of GNU Autoconf is that it runs on a wide variety of systems.
On the other hand, many programs are for in-house or personal use and will not run on obscure systems. The cost of writing for portability simply might not be worthwhile in these situations. And that’s ok, notwithstanding some conventional wisdom of “always try to be portable.”
That is, test and [ are specified by POSIX and usually are physical binaries (but might also be masked by shell builtins). Whereas [[ is not specified by POSIX and usually only exists as a shell builtin.
This to me seems like good practice, it's better to use a 'real' programming language when you're crossing the boundary from 'script' to 'program'
Shellscript has way too many idiosyncracies and weirdnesses that would have been beaten out of a proper programming language by now. (I know that talking about weirdnesses is amusing in relation to perl which also has a whole armload of them.)
No, power shells problem is that it cannot decide whether it's a programming language or a shell REPL, and ends up doing both badly. The syntax is too cryptic (and far too verbose) for a REPL, and too "shell-y" for a PL.
bash, for all its many many many flaws, is quite clearly a REPL first, PL second.
Not installed by default on BSDs. Sometimes not installed in minimal environments, e.g. where BusyBox is all you have available. Maybe not installed by default on Unixes? Not sure
When your script will be running on customer machines, you often don't have the luxury (and sometimes it's not technically possible because it doesn't exist) of installing bash into the environment.
These scripts are often used to install the application, so that introduces a chicken-and-egg situation. How can I install bash before running the installer?
Also, we'd have to build and maintain our own just for platforms missing it, which increases cost and complexity. It's easier, cheaper, and less error-prone just to use a minimum common denominator that runs on everything.
Additionally, when your customers have strict policies regarding how applications are approved for use and have to vet and test everything you give them, it's best practice to avoid installing anything that you can avoid installing. Doing that minimizes the effort and hassle for both the customer and the developer.
How can you install bash without using it? With sh as bootstrapping shell.
Build and maintain your own bash or install script? You don't have to maintain it, it's just building which is already provided by original source build scripts. You can install it as "mysolutionsh", nobody cares as long as you link it to PATH or reference it explicitly. Still easier than avoiding bash everywhere but in personal setups. Yours already familiar with it, that's a waste!
That's the sort of yak-shaving I will never do. I'd rather just not use such a primitive unix and/or be in a job where one of pdksh/ksh/zsh/bash is not available at all.
Well it’s good that you have the choice to; but it’s also not all that difficult to understand that not everyone who needs to use your script, has that choice.
That's cool. We all decide for ourselves what markets we're willing and unwilling to address. But the market that involves "primitive" unix is lucrative and that can be worth it.
For me as a developer, it's not a big deal. You can do all the same things, just in a slightly different way, and you have a script that can run almost everywhere.
It's not usually hard to do, no, but it is >130K of C alone, excluding whitespace, comments, tests, examples, etc. I don't want it on my system from a security perspective alone.
Add in the bootstrapping pain it imposes due to autoconf and other stuff, I think there are many valid reasons to avoid it and choose another shell that is more auditable yet still has just as many eyeballs on it (e.g., mksh - the default on Android; dash - default on Debian; BusyBox - every embedded system).
Writing Bourne-compliant scripts ensures maximum portability.
As many here have noted, bash isn't universally available, with another possible issue being OpenWRT devices. Stock/base images tend to use a Bourne-compatible shell, not full Bash. Though the latter's installable through opkg, for sufficiently small devices (typical of consumer kit), you simply won't have the space to install them.
There's also the slight PITA that Apple's OSX ships with a very old, pre-GPLv2 Bash, out of licensing concerns. (Apple is phenomenally averse to GPL-based code, much as some *BSDs are, such as OpenBSD.)
And if you're dealing with legacy systems (which tend to be extraordinarily and stubbornly persistently legacy), you'll often find that either bash isn't present or is quite dated.
I freely confess that I tend to write fairly recent-feature bash scripts myself by default, and appreciate many of the newer features. But if and when I am writing portable code, I'll go back to Bourne-compatibility.
But when writing system level code, an appreciation for standards and the very long tail of legacy standards and the limitations they impose is in fact a large component of professional maturity.
I've worked on several modern projects that needed to be able to run on a wide variety of Unix platforms, several of which didn't have bash. Writing for the common shell denominator was important, not archaic.
I wouldn't say it's an archaic problem, an example I can think of is writing scripts for infrastructure running minimal docker images where you want to keep the image size to a minimum, you would usually need to support both bash and other shells.
Well, I am a professional who administrates quite a lot of systems.
And they all run bash. They all understand `[[`, they all understand `~=`
If any of them did not, then I'd ensure they do.
Yes, shells that do not support modern bash syntax ARE archaic, no matter who sits in front of the screen. And same as I consider it part of my job description to keep systems up to date with security patches, I consider it part of my job to ensure they use modern versions of contemporary and widely used scripting languages and shells. And that includes bash.
So no, relying on features that can be expected on a modern system is not a sign of inexperience. Same as a dev is within his rights to not write his python code under the assumption that it has to talk with a python 3.4 interpreter, he is within his rights to not expect a system that only knows sh.
Eh, there are things that shell is really good at, and there are things that other languages are good at. I will grant that shell is best for glue code; my personal heuristic is that I stick to POSIX sh, and if that hurts then I take that as a sign that I should be considering moving to Python or whatever. But avoiding it completely strikes me as a poor choice, because for the "glue together separate programs and manipulate files" tasks that it's meant for, it's really good and IMO everything else still falls short.
it’s contextual as most things. need to actually use and work on the machine? use whatever shell you want to make your life easier.
need a script to run on many different systems and/or need to write a script to be managed automatically by a service account? probably you want a shell with syntax that is guaranteed to be the same on all your systems.
> Aside from it being bash (a great reason not to use it as far as I’m concerned) it’s now a 17 years old version of bash.
I thought people liked macOs for its vintage feel? Remember a time when computers could only render a single menu bar in a fixed location, feel the experience of SYN floods, run a version of bash that is old enough to vote in the next presidential election.
Honestly this is such a waste of time every time I have to argue with developers about installing up to date homebrew (or whatever) versions of the coreutils that I wish Apple would simply DELETE all these ancient versions of tools from MacOS.
As a bonus, homebrew does not offer --with-default-names any more and some tools (like make) are posted into different paths so you need to add your own symlinks or add multiple paths to your PATH.
According to the interwebs, zsh became default with Catalina in 2019 [1]; ten years after bash 4 was released with gpl v3 or later.
Also, the interwebs suggest Apple used to use tcsh as the default shell[2]; I don't know when they changed that, but it may have been after bash 4 released? (Thanks, 10.3 was in 2003, so several years before the license changed)
If you want to nitpick, it was first done by Ubuntu and then later upstreamed into Debian (from which other child distros, as you correctly point out, inherit it).
I really didn't understand why the last if statement is confusing. Is it because when starting out with shell scripting one would usually assume that the [ is a part of the bash scripting language not just another program? If it's then I think I get it now. Otherwise please mention why it's surprising. Also, @author thanks for a nice article. was a good reed.
I have strong opinions about shell and they don't actually line up with the rest of the world...
I believe that [ should never be used, only "test" because [ gives the illusion that the mechanism is some part of the language syntax when it is just another "program" (I'm including built-ins and functions in "program").
(if / || / && look at exit status; a program can't see the exit status of other things other than looking at the magic variable $? which is just another string once expanded; case looks at strings but doesn't operate based on exit status and doesn't set an exit status as part of the case ... esac operation; "programs" set an exit status)
I also believe that [ / test should only ever be used for evaluating filesystem constructs -- test -f /dev/null and if you've got string evaluations use case.
Unsurprisingly, most scripts make me itchy, and scripts that I write people find weird.
[edited to add the explanation of "program" vs "syntax" opinion]
To emphasize that the thing after the if is just "look at the exit status of the last command before the then."
if
ls /tmp/goober
test -d /tmp/goober
echo "I'll always execute the then clause because echo will always return a 0 return code"
then
echo "this always gets run"
else
echo "this never gets run"
fi
because the "if" is just looking at the exit status of "echo".
largely agree, but stylistically speaking I prefer the following:
if ls /tmp/goober
test -d /tmp/goober
echo "I'll always love you or whatever"
then echo "the grasshopper always jumps higher"
else "never gonna let you down"
fi
i.e. all the commands line up at the 6th column, leaving the if/then/elif/else/fi stuff appearing on the 'margin', as it were.
This works particularly well, visually, when nested ifs are involved.
Hello fellow traveler. I've got 8 years of scripts with `test` in them to prove I agree with you. It's a lonely road out there... I blame the Google shell style guide.
I picked up the habit of preferring `test` when I was writing scripts that needed to run in both `sh` and `bash`, but I kept it because it makes more semantic sense to me than treating a character like `[` as a command. It's also weird that `]` is an argument to `[` rather than also being a binary. I mean, I understand the technical reasoning... but it feels like a hack.
Hello fellow traveler. I've got 20 years of scripts with `if test` in them. I only use `[[` when I need functionality it provides that `test` does not (pattern or regex matching, typically, and for pattern matching I'll generally use a case statement instead).
Yeah that's when I use it too. I figure if I've already given into the temptations of Bash, I may as well go all the way. Sometimes if I'm feeling extra frisky I even do [[ .. ]] || { echo "error!" ; exit 1 }
The Google style guide doesn’t disagree with you, it says to code exclusively for bash where possible (ie mostly everywhere inside Google) and to prefer [[ over [ and test:
As someone who enjoys shell scripting I don't think I have ever used the "expr" command. this astonishes and baffles me because I used to read the man pages for fun, I even had a little script to pick a random page. So thanks a bunch for bringing it to my attention.
My only guess as to why I am unfamiliar with the command is that perhaps younger me failed to figure out "expression" means regular expression. It only clicked this time because of your comment and the see also: re_format link at the bottom.
Yeah it's slightly obscure for a lot of people I think: most who do know of it seem to associate it purely with arithmetic, and rightly use `$((...))` instead.
The fact that the regex is written bare and unquoted, got me a while ago. I'm someone who religiously quotes everything; it took me a while before I figured out why my stupidly simple regex doesn't match.
For substring matches, [ and * globs are generally good enough.
[ "$bar" = extra* ] && echo '$bar began with extra'
Bash's regex dialect is primitive and rarely worth fussing over. For anything complicated, use another tool be it grep, awk, perl, or such. There are diminishing returns of obsessing over doing everything in bash when a complicated task demands more capabilities suitable to another tool with greater reusability, modularity, and intrinsic types.
Bourne has said there were proposals from the other users at Bell Labs to use braces or brackets with the "test" form, but these suggestions were not adopted. Bourne has said he did not like having to worry about missing braces in C, and there were no IDEs with automatic syntax-checking like today, and that is why he used macros to allow him to write in C using Algol-like syntax. That's what he said. Maybe that's why he decided not to braces or brackets with test.
Personally, I prefer the "test" form to the "[" form. It's easier for me to read. Personal preference. To me, "[" looks uglier.
One place where you learn not to use [ or [[, or be very careful about them is autoconf shell scripts (and that applies to more than `if`, like globs, sed, awk, etc.). Because square brackets are quote characters in autoconf. https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/a...
Interesting... Doubly so because I thought busybox was a sort of linux crunchgen, A way to pack many independent executables into one to save space. With crunchgen at least, each executable name is then linked to the packed binary, that is, hardlinks. and the filesystem name is used to pick the correct code to run. Why did they go with softlinks? my guess... It can be moved across filesystem boundaries. Perhaps interference from cgroups?
Both just look at the first entry in argv. Whether you used a hard- or symlink isn't very important for that. Or at least, that's how FreeBSD crunchgen works. Maybe they changed that in OpenBSD?
What bothers me is that you can write equivalent code with `test` and `[`, but when using `[` you need to terminate your conditional expression with `]`. Why? Isn't it the same binary? Why is the shell adding this extraneous requirement, just to surface "syntax errors" rather than simply treating `]` as a no-op?
# file /bin/test /bin/[
/bin/test: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=6fe552c80ab0b3d3e60de2ab09167329e222eb67, for GNU/Linux 3.2.0, stripped
/bin/[: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=99cfd563b4850f124ca01f64a15ec24fd8277732, for GNU/Linux 3.2.0, stripped
#/bin/test --version
#/bin/[ --version
[ (GNU coreutils) 8.30
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by Kevin Braunsdorf and Matthew Bradburn.
if some_command 1>/dev/null 2>/dev/null; then
: # A-OK
else
here_is_the_code_i_actually_needed
fi
And it leaves me wondering if there is a way to “negate” the exit status of a command...
(The command was “docker volume inspect VOLUME”, and in one place in my script I had to do things when the volume existed, and in other places I had to do things when the volume did not exist...)
No, it's part of the shell. Do "man bash" or "man sh", then search with "/\!". From man bash: "If the reserved word ! precedes a pipeline, the exit status of that pipeline is the logical negation of the exit status".
> And now for the final lolz. I’ve said above that these are the commands you use to evaluate expressions… but the shell also has expressions of its own via the !, &&, and || operators—all of which work on command exit statuses.
It works for simulating very basic boolean expressions, but it gets ugly quickly if you need some more complex combos of NOT, OR and AND. I wish Bash had proper boolean expressions support.
The syntax is a bit unusual coming from more modern languages, as is the use of 0 as true. But you can express whatever conditionals you want. Remember that C originally had no dedicated boolean type either.
You could also use numerical expressions although I don't recommend it due to being even less readable (even more so if you have expression complex enough that you need to protect against overflow):
baz=$((foo + bar)) // foo && bar
baz=$((foo * bar)) // foo || bar
Instead of trying to make a shell script portable across all different shells, I prefer to just use #!/bin/bash and not worry about it. (Of course bash version compatibility is a concern of it's own)
I'll never understand how there's a whole class of developers who absolutely despise JavaScript for acting weird when adding arrays to objects, but at the same time gladly write bash scripts and send each other articles about how [ is totally a program but ] isn't and that's all fine and dandy and in no way objectionable.
EDIT: this comment is a bog standard HN middlebrow-dismissal and the blog post doesn't deserve this for to be the top comment. It was morning and I was grumpy. I can't delete it anymore so would appreciate some downvotes.
Well, shell scripts are small helper programs, where you don't have to implement all sorts of business rules and complex object interactions. I am pretty sure most developers (I definitely) would be horrified at the prospect of having to to implement in shell what we are now doing in JS. Even if that were possible.
The likes this gets is the design simplicity and uniformity (across processes), not how it looks and what it actually does.
Oh come on. Who are you to say developers don't have to implement business rules and complex interactions? You're just making an after-the-fact rationalization for a terrible, foolish, thoughtless, pointlessly complex design. And then trying to dictate how people should use their tools and what kinds of problems they should limit themselves to solving.
It's not like the shell script designers sat down at a meeting and said:
"OK, it's very important that we don't want people using this language to implement all sorts of business rules and complex object interactions, because decades from now there will be invented an Ousterhoutian dichotomy and government regulations enforcing that developers should use other kinds of languages for that, because of the essential definition of what it means to be a shell scripting language, so we've got to come up with some way of punishing people who attempt to do that, and introduce obscure hard to spot bugs in their programs as a consequence if they have the audacity to do that, or even if their initially simple scripts later get more requirements and have to become more complex. Now let's brainstorm about how we can do that, and make sure the syntactic syrup of ipecac we come up to solve this problem fits in well with the rest of the language design by being totally off-the-wall and unlike every other piece of syntax in any other programming language including itself."
Then again, maybe you have a point, and they did do it on purpose, judging by how terrible the rest of the language is!
"Language Design Is Not Just Solving Puzzles" -Guido van Rossum
>Summary: An incident on python-dev today made me appreciate (again) that there's more to language design than puzzle-solving. A ramble on the nature of Pythonicity, culminating in a comparison of language design to user interface design.
>Some people seem to think that language design is just like solving a puzzle. Given a set of requirements they systematically search the solution space for a match, and when they find one, they claim to have the perfect language feature, as if they've solved a Sudoku puzzle. For example, today someone claimed to have solved the problem of the multi-statement lambda.
>But such solutions often lack "Pythonicity" -- that elusive trait of a good Python feature. It's impossible to express Pythonicity as a hard constraint. Even the Zen of Python doesn't translate into a simple test of Pythonicity. [...]
>Guido: Language Design Is Not Just Solving Puzzles
>And there's the rub: there's no way to make a Rube Goldberg language feature appear simple. Features of a programming language, whether syntactic or semantic, are all part of the language's user interface. And a user interface can handle only so much complexity or it becomes unusable.
>The discussion is about multi-statement lambdas, but I don't want to discuss this specific issue. What's more interesting is the discussion of language as a user interface (an interface to what, you might ask), the underlying assumption that languages have character (e.g., Pythonicity), and the integrated view of semantics and syntax of language constructs when thinking about language usability. [...]
Ousterhout's Dichotomy is a contrived descriptive not prescriptive fiction, to rationalize the design of TCL after the fact. And despite its flaws and limitations, TCL is orders of magnitude better and more thoughtfully designed and purposefully thought out and internally consistent than any Unix shell scripting language.
>Ousterhout's dichotomy is computer scientist John Ousterhout's categorization[1] that high-level programming languages tend to fall into two groups, each with distinct properties and uses: system programming languages and scripting languages – compare programming in the large and programming in the small. This distinction underlies the design of his language Tcl. [...]
>Criticism: Critics believe that the dichotomy is highly arbitrary, and refer to it as Ousterhout's fallacy or Ousterhout's false dichotomy.[4] While static-versus-dynamic typing, data structure complexity, and dependent versus stand-alone might be said to be unrelated features, the usual critique of Ousterhout's dichotomy is of its distinction of compiling versus interpreting. Neither semantics nor syntax depend significantly on whether a language implementation compiles into machine language, interprets, tokenizes, or byte-compiles at the start of each run, or any mix of these. In addition, basically no languages in widespread use are purely interpreted without a compiler; this makes compiling versus interpreting a dubious parameter in a taxonomy of programming languages.
Well, I agree with all of that and I think maybe my comment was misunderstood.
I was responding to the parent comment, saying "why do people like this". Its minimalism is ok-ish for small programs where you don't feel the pain so much and kind of beautiful in its solutions of packing everything into a separate executable.
And I absolutely stated that we have to implement the business rules and that I would be horrified to implement it in such a language. Therefore, yes, the language comes from a different age and shows it and, yes, would never consider it for any complex thing.
I dislike JavaScript and shell scripts equally, but sometimes I have to add a feature to a web page and I have to use JS, so I do, and sometimes I need to automate some un*x system task in a portable way and without heavy deps and shell scripts are the obvious solution.
What annoys me is using JavaScript and shell scripts when there are clearly superior alternatives and no clear advantage for it besides the familiarity (which, admittedly, can be a strong argument).
Shell scripts being an arcane mess is no excuse for Javascript being as clunky as it is, and vice-versa.
One clear advantage that both JS and Bash have over nearly every other language is stability. Code you write today is very likely to still work in 20 years.
You can replace bash with JS. Scripting is their common application domain.
Can you replace JS with bash, though?
If the answer is "no," that means JS has applications (e.g., servers and web clients) that bash can't be used for. And they're saying JavaScript is bad at them. Bash is arguably worse, but it's usually not an option in the first place so you don't get to complain.
Noo-ot really except in a Turing-tarpit sort of way. Like it or not, once learned (!) Bourne shell together with the traditional tools is a well-designed user interface and works even better for that than, say, Tcl[1]. Not an automation or scripting language, a user interface for all daily interaction. I would absolutely hate to manually sort through my files in JS, while in shell I can often do it faster than in a GUI file manager.
And, of course, it’s also a fairly strong contender as a programming language in the paradigm of many pipelined imperative processes—probably because that paradigm remains largely unexplored. I can only maybe name Icon as a viable competitor, and Icon’s also very nice. (Python, no matter how “inspired” it is by Icon, has traded its command of streams for more mainstream ease of use.) By comparison, the old complement of parallellized JS build tools (Gulp? Grunt? I forgot, it’s been a while) always surprised me with how awkwardly it accomplished shell-script-equivalent tasks.
To be clear, it’s not that Bourne shell is good and JS is bad. JS is a passable Fortran[2], while shell is at best a marginally functional one—there’s a reason Awk exists. But shell competes in categories that most other languages don’t even try to qualify for.
[Yes, I know about rc. I happen to think rc’s focus on one-level lists of strings (incidentally shared by Jam, an attempt at a better make) is a mistake, and more consistent arbitrarily-nested quoting, giving a “stringy Lisp” in the vein of Tcl, would be the way to go.]
> it’s also a fairly strong contender as a programming language in the paradigm of many pipelined imperative processes—probably because that paradigm remains largely unexplored.
The paradigm is called point-free programming[1], right? I don't know how "imperative" is relevant here--you can rewrite most of the coreutils in Haskell, for example.
> Noo-ot really except in a Turing-tarpit sort of way.
That means bash has applications (e.g., batch processing and file management) that JS can't be (easily) used for. And they never said bash is bad at them. That...proves my point?
(I am going to use the word "shell" to refer to posix-sh like shells specifically)
I think a lot of this is a (li)nix culture thing.
Not liking javascript is "cool" in some sense. It's this new fandangled web language, not a "real" programming language in this culture. Shell scripting is not seen in the same light because its older and associated with unix I guess.
Not liking newer technologies in general is a thing that I have noticed, like with Rust or even C++ which isn't that new!
I do think shells have a better excuse for being bad programming languages than javascript though. Shells are primarily an interface to your computer and not a programming language. I use a shell in a terminal emulator to do most things on my system, like managing files and updating the system, you couldn't easily use javascript for this without writing a shell in javascript.
Shells are also much older than javascript, and "newer" shells like bash or zsh need to maintain some level of backwards compatibility.
Mostly shell scripts are used for very small tasks and not for writing large programs which is another factor.
You do bring up a very interesting point though. I think this idea applies to perl as well. It's interesting to see the difference in how these things are viewed.
one language being the only option available on browsers and the other being a (replaceable) glue layer between _other_ programs? I also see minimalistic beauty in the fact that `[` is not even part of the shell. "do one thing and do it well" at its finest.
Writing bash scripts seems like the ideal use case for Chat-GPT style programming.
The programs are fairly short, examples are common in the corpus, and the syntax and execution model are so inscrutable that only a machine can pretend to understand what’s going on.
I tried doing this and it made a bunch of mistakes with handing spaces and whatnot that makes shell scripts often brittle. Of course, a human would probably make the same mistakes, because who actually knows how to do that correctly? But it doesn’t really make me comfortable using it for anything but the smallest of automation tasks.
I already tried to customize my command line prompt with it. All the arcane colouring character decodings and random character escaping missions became a "put git branch names in parenthesis and make it cyan".
> Some people insist that Linux distributions be called GNU/Linux, not just Linux. The reason is that, strictly speaking, Linux is just a kernel and, when you are using a Linux system, you are primarily interacting with the GNU userland. While I sympathize with that idea, I’m not the one to adopt the GNU/Linux name, in part because most Linux distributions today ship with software developed by many more vendors than just GNU. In this post, however, I use the GNU/Linux term because saying Linux alone would be unfair to Linux.
Bash is "real language" if you treat it as such. Yes, there are plenty of spaghetti scripts out there, riddled with global variables and full of side-effects, but if you write shell in a principled manner, you can solve some pretty big problems with ease, and you can write maintainable code.
I wouldn't start new projects in shell, of course, but one area in which I think the shell shines is in writing integration tests for tools. More on this in a recent post I wrote: https://jmmv.dev/2023/10/unit-testing-with-shtk.html
> Bash is "real language" if you treat it as such. Yes, there are plenty of spaghetti scripts out there, riddled with global variables and full of side-effects, but if you write shell in a principled manner, you can solve some pretty big problems with ease, and you can write maintainable code.
Even carefully written bash code tends to be much less maintainable than code in better languages. It's just badly designed on multiple levels, like the famous post about PHP. Yes, if you're really careful you can write decent code in Malbolge - but why would you?
> I wouldn't start new projects in shell, of course, but one area in which I think the shell shines is in writing integration tests for tools. More on this in a recent post I wrote: https://jmmv.dev/2023/10/unit-testing-with-shtk.html
The fact that you've written something that compiles to shell scripts rather than writing shell scripts rather undermines your claim that shell is a decent language. Integration tests of tools are important, but I'd still find e.g. TCL a much better way to write them.
> but if you write shell in a principled manner, you can solve some pretty big problems with ease, and you can write maintainable code.
So if you are extremely good at writing shell, and you write it extremely carefully, then you can achieve basically what you can do in other languages without those caveats?
It sounds like we both agree that shell is not a sensible choice for scripting, for most people, then.
The same can be said about QBasic as well, honestly. And here's the catch: people who are inclined to write anything in a principled manner are likely the ones who'd write things in a principled language instead of e.g. shell. Or to put it in another way, if someone chosen to write in shell (or failed to consider an alternative, which also happens quite often), they're quite likely exactly the person to not write anything in a principled manner.
I think one of the Rust's creators (or was it Go's?) said that they wanted to get the C++ developers to switch to their language but that didn't happend, they unexpectedly got Python developers instead but retrospectively it makes sense to them: people who wanted to switch from C++ and could afford to had done so already, so the C++ developers are the people who either don't mind C++ or the people who have to write it.
Which brings me back to my point: yes, technically you can write decent code even in a sloppy language but that misses the bigger picture which is that most of the code written in a sloppy language will inevitably be sloppy because most of the people who write it won't care, due to the dynamics described.
Heck, the Ops department in my org was forced by the security guys to globally enable ShellCheck on commit for their internal repos, and those security guys still have to drop by about every 2-3 months to rip away their "# shellcheck disable" pragmas and force them to actually fix their broken code because those people in the ops actively don't care. The Ruby scripts they write end up slightly less broken because Ruby itself is a slightly more principled language, not because they suddenly care more when they write Ruby.
Not to mention myself: I've spent quite some time and energy on learning shell's semantics and tricks and quirks and DOs and DONTs but it's such an infinite, never-ending descent into abyss with rewards of dubious value that nowadays I just don't care. Whenever I have to write a one-off script, I give up even before I start and write it however sloppy, with minimal quoting, to save my time; and only when it breaks, or when I have to reuse it, or when I have to share it — which happens quite rarely — then I re-write it in a proper language. On the whole, that definitely saved me both my time and my sanity. As for the cases when I have to write properly behaving shell script, well, those are almost arise in the course of the tasks that can be delegated to our ops team and now that's their problem.
P.S. I found that re-writing naive but broken shell scripts in a proper language is quite easy: the intended semantics is generally obvious, it's just the shell that actually requires quirkier syntax to propely express it; re-writing the (mostly) non-broken shell scripts is much harder: you have to decipher the intended meaining from the quirky syntax while keeping in mind that the original author still could have gotten it wrong by not knowing about a particular quirk you're aware about (or vice versa).
I agree. If a POSIX shell script serves your needs, use it. If it doesn't, you shouldn't be reaching for a more powerful shell, but an actual language. I'd reach for Perl before Bash, and I can't stand Perl.
People would had such a take and tell to use perl twenty years ago. Looks where we are now. I'd rather have improved my shell scripting abilities right away.
It's a three (optional) steps thing really:
- I want something that can use on multiple systems right away (shell)
- Ok things getting a bit serious, I'll allow myself to add some pre-requirement tooling on the system (others script)
- pre-requirement sucks, I'll invest in native code tooling and produce binary (eg rust & go for the choices of the moment).
Agreed. If your shell script has more than the most basic of functionality, you should probably write it in literally any other language. Like another poster said: I don't like perl, but I'll take it any day of the week over bash.
Here is something related from 2021 that also touches on bash's [[ operator and that I think you might enjoy in this context: https://jmmv.dev/2021/08/useless-use-of-gnu.html