I spent many years avoiding shell, in large part because all the arbitrary-looking syntax and features (like quoting rules) the speaker highlights make it frustrating and difficult to master, and I felt like it was a treacherous enemy to be overcome through desperate struggle, rather than a tool and ally to rely on. I know some people thrive on mastering complex and intricate rules, but that's not my forte. I do better with overarching principles and a unified set of orthogonal constructs, such as C (ok, debatable, but compare to C++), Lisp, Python or Haskell.
It took a lot of pushing from a mentor for me to finally click with the overarching principles of shell and see how to master it and become productive with it. He really forced me into it as well, I wasn't a willing pupil for quite some time, but he kept persisting and finally I started listening to him. I'm so glad I did. I've never read any book or tutorial or guide that teaches what he taught me either, despite searching several times. Also, I don't think I could have learned to master shell if I hadn't learned functional programming first. The day I realised how amenable shell is to higher-order thinking was a real epiphany.
These days shell is a really important tool in my toolkit, and I can see just how much some of my colleagues are held back by not knowing it.
Anyway, I really enjoyed the video, it was a nice exposition.
I agree on both counts. My first experience with shell was the fact that 'foo=bar' works for assignment but 'foo = bar' doesn't. Surely that must be a mistake! Why would anyone design such a silly language?
I got past all of that, and I also see people struggling to accomplish things that can be done in 30 seconds with shell. My main language is Python, and for certain tasks it just can't compete with shell. Believe it or not I think I once wrote a Python script to do what "find | sed -i" does.
Now I'm back in the "who would design such a silly language" mindset, and I am designing and implementing a new shell. You might like some of my blog posts:
"Pipelines Support Vectorized, Point-Free, and Imperative Style"
Thanks for those links, I think I will enjoy them. I was actually thinking about Forth as well when I was writing those comments about functional programming. I'll be interested to read your take on it.
Yes, totally agree. But it offers something different to say Python, as the video points out. Some things are more compact in shell than in Python, because it's not the same sort of programming language as Python. And I reach for Python as well, when it's the right tool.
True. I'm not against a DSL for working with CLI programs, pipes, output streams, processes, etc. Just that the shell as we know it is not the optimal (or even close to optimal) example of that.
Something like ZSH and the modern Fish shell showed how much traditional shells can improve, but we also need some better primitives, and more controlled experience.
Oh, and they should NOT be based on emulating 40+ year old teletypes anymore, except as part of a "legacy" mode.
Totally agree, the concept is nice and really useful but its implementation is for a large part really a big pile of fragile and confusing hacks. Most of the power comes of powerful tools like find, sed, awk (being a programming language on its own), etc. The language gluing them together is nicely compact but apart from that very disappointing. Also the everything is a string approach has its limitations.
A python script may sometimes be longer but a lot less riddled with annoying corner cases.
I'm the opposite, I avoid GUIs. I use pretty much only the shell, and text-oriented programs. I use a graphical desktop just to manage my terminal windows. The only other application I use daily is a web browser.
I guess maybe because I started with computers before there were GUIs that is what felt normal.
> I guess maybe because I started with computers before there were GUIs that is what felt normal.
For me, probably not. I did use DOS before Windows as a kid, but that was long before really learning to use a shell. Familiarity with DOS simply made me familiar enough with the idea to be unafraid of it.
When I learned to manage files, handle software, etc. I learned to do it with a GUI in Windows. When I started to use Linux, I was vaguely aware of the usefulness of a shell, and took the opportunity to really learn. To this day, I rarely even have a file manager installed, in favor of a shell environment.
Ha! I should have been more precise in my language. I was avoiding learning how to write more substantial shell scripts and programs, not the command line per se.
I also got my start on pre-GUI computers (a VIC-20, specifically).
I'm confused. You had a good time with C but trouble with shell? I was strongly under the impression that 90% of Unix shells were strongly inspired by C semantics. Return codes, quoting, strings in general, etc. You maybe could even try csh if that was better.
Every language has its warts, but C is a pretty small language when it comes down to it. Compare that to just the number of different ways you can expand a parameter in bash:
That's the sort of thing that turned me off. I still tend to avoid those nooks and crannies, although they are sometimes useful. I'm not an expert.
I'm not sure about Unix shells being inspired by C semantics. I actually started out on csh or tcsh, I forget which, two decades ago. I'm not sure that making a shell language "C-like" is a useful goal, and those two shells both fell out of favour because they have many flaws. I switched to bash quite quickly.
- I used to spend a lot of time stuffing things into variables and then trying to operate on those, which is how you do things in imperative languages. Shell is better thought of in dataflow terms, i.e. pipelines. So I started using pipelines more and more.
- The corollary is that you start thinking more about data (passing between pipeline stages) and less about control flow.
- Compare an if statement in C, for example, to an if statement in a functional language. In C an if statement has no value per se, but in functional languages if "statements" are actually expressions that have a value:
f :: A -> String
f x = if isFoo x then "foo" else "other"
In shell an if statement is just another command, so you can do things with its "value", i.e. its stdout:
if do_test
then
cat file1.txt
else
cat file2.txt
fi | grep pattern
Because I was so used to thinking of if statements as just control flow, I never used to think of piping them to another command like that.
Also if you think carefully, that last pipe to "grep pattern" has the flavour of partial evaluation in functional programming.
- I tend to think more in terms of building up an execution environment these days, in the same way that lexical scope works in regular programming languages. In shell, of course, that means command invocations, sub-shells, etc., because the shell doesn't really have good lexical scoping. The simplest example I can think of is executing some commands in another working directory with a sub-shell:
(
cd $dir
run_test > "${f}.log"
)
You could compare it to RAII in C++ or with- style macros like WITH-OPEN-FILE in Common Lisp.
Those are just some examples.
EDIT: Also, have a read of the links in chubot's post: https://news.ycombinator.com/item?id=14012453 . That's the sort of mentality that will change your approach to shell programming.
This[1] Stack Overflow question sums up everything I dislike about shell scripting. Not only does it highlight the finnicky string-oriented nature of it, but the answers...
Multiple contradictory answers, disagreement, sub-sub-sub explanations, historical context, caveats, workarounds, hypothetical scenarios, portability, personal taste, different versions and implementations all show up. One answerer felt the need to include a footnote.
It's like the opposite of Python's "there's one right way to do it": There are 50 right ways to do it, all of which are wrong in some case or another, potentially depending on which way the wind is blowing.
Not to start a war, but Python's "there's one right way to do it" is nothing more than an ambitious claim with little ground. Maybe it applies for doing something brainless like printing a string, but the claim completely falls apart when trying to do something as simple as list indexing/slicing or even iteration. I'm not condemning Python in anyway, I am a fan of the latter mention features, but that "zen" of Python is not applicable.
I for one like my shells dirty with hidden corners and black magic. Name anywhere where you do not have bash available; embedded, grub, rescue boot shells. I don't do embedded. Never will.
But on a serious note, does anyone know a good resource where I can pick up a better la guage and never have to bash script again? I love writing it. It I am sooo restricted by it, yet keep solving problems with it and also making good money like last night. Maybe Python is the way?
I've been down exactly the rabbit hole you're speculating about, and my firm conclusion is: python is not a shell replacement. [update: some other posters here are referring to xonsh. Maybe they succeeded where I failed, and python can be done shellable. Will be looking into this.]
* Python does strange things with IO buffering. Just when you think you've nailed all the edge-cases of this, some more come along. Its decisions are good for a general-purpose programming language, and awful for a shell.
* Python is awkward for some close-to-the metal operations such as interaction with unix sessions and process groups.
* Python forking is wacky. There's magic in there, and there's some stuff that gets inherited under the cover. If you fork and don't immediately execv, you are probably in for Interesting Times.
When I looked at the code for Ansible, I felt a bit like I'd encountered a rival alien civilisation who had tested and reacted to the same laws of the universe, and reached identical conclusions. Ansible is written in Python. But it generates bash scripts, and then bash does dispatch.
Given the power of general-purpose languages we have on unix, it's remarkable how stunted shell evolution is. Which is to say, there is an exciting opportunity here.
A shell built as a Forth dialect could be a game-changer. It would have a certain number of powerful builtins. It may even evolve higher-level programming concepts. I am interested in further discussion towards this, and maybe a new github project, particularly a collaboration with someone with high fluency in Forth. I'm in London: beers.
The art of a good shell would merge (1) fluid interaction; (2) effective wrapping of the unix API; (3) high-enough-level programming.
I'm familiar enough with Bash to have written an opinionated but more-or-less comprehensive guide to it. I think Bash should be avoided as much as possible. Bash is great for things that you would have had to type out anyway, but using it to write programs is foolish. My rule of thumb has become, "if it's more than 30 lines long, or if it uses more than two variables, it should be written in another language". That said, I'm in the process of learning how to do that myself.
Python is definitely a capable language for scripting, but I would also point out that Homebrew is built on Ruby and uses that to great effect. You might also check out this article on scripting with Ruby:
Bash (and other shells) have their purpose, and that purpose is running processes, and process-, filesystem- and general-system administration. Pipes, especially, are so much easier in a shell than any other way that I have seen (I just noticed ninjin's comment about Julia - I will have to give it a look.)
For this sort of thing, I will use a combination of Bash, Awk, Perl and Python, with Bash as the foundation, but often using at least one of the others. For example, if a step in a pipeline can easily be solved with regular expressions and/or simple arrays and hash tables, I will pick Perl, but if it needs more complicated data or functions, I will pick Python.
I appreciate your point, but I think the choice to write all or parts of an application in shell depends a lot on what you need to achieve. Writing 100 lines of shell to glue parts of your application together may be the best way to ship something today, even if you plan to replace that with another language in 3 or 6 months time.
Would you care to give a link to your guide? I'd be interested to read it.
Commentary, contributions, and criticism are invited.
As far as 'shipping something today' is concerned, that's what keeps me from learning vim. At the moment I'm both working and going to school full-time, so I feel like I can't quite justify the time investment, but I know I'm holding myself back from being a far more effective programmer.
I also have an acquaintance doing computational biology for whom Bash is an essential part of his toolkit for DNA analysis. Bash sometimes is the right tool for the job. It excels at text processing. It's just archaic and ugly, and we have nicer options available for programming languages.
(and now that my SO is up I think I'll watch this video)
> 'shipping something today' is concerned, that's what keeps me from learning vim.
If you want to learn vim (I would definitely recommend you do), learn a little at a time. Vim is better suited to this approach than you might think.
People tend to focus on vim's learning curve, but as soon as you grok 3 things, you can do everything you used to do with normal editors. Then all you need to do is learn one small function at a time.
1. :command
:w save the file
:q quit vim
2. Normal mode
You are here. Pressing a key runs a function. Escape cancels. u is undo, Ctrl-r is redo.
Every key has some specific use. You can learn them one at a time.
My favorite is . (the . key). It repeats the most recent action at the cursor.
3. Insert mode
Pressing i puts you in insert mode. Here you can enter text like normal until you press Escape
Remember that everything that happens between pressing i to pressing Escape is one action that can be undone or redone.
Later you will learn that a s r and o all do similar things to i.
Yes, that line came out of arguments with greybeards about the problems of the younger generation and using all these fancy frameworks and wasting memory and CPU cycles.
I will also add the further caveat for anyone else who arrives here that this was written in stolen hours in the last two weeks, and specifically because the page of 'Assumed Knowledge' referred to in the introduction seems to have been the only instruction in the shell given at Epicodus' Intro to Programming. I'll never love Bash as a programming language, but given that it's a tool developers use every day, I do think there's a certain minimum amount of knowledge required to use it well.
While Python does the trick just fine, I find how Julia handles "shelling out" and IO with the shell to be very pleasing. I am probably still a bit ignorant in terms of what is out there, so I am reading the comments with great interest.
Python and Perl are both reasonable substitutes. Perl is closer in spirit (black magic, direct call of external programs with backticks or qx(), almost direct translation of sed and awk calls).
Apart from shells having wacky syntax and semantics, due to shell pipelines being plain text, you spend a lot of time writing flaky mini parsers with stuff like sed/cut/awk/perl one-liners/ to get the data you want.
In an alternative universe it would be nice if all these tools would consume and emit s-exprs. Or maybe I'm just wishing for the return of the Lisp machines, where AFAIK "the shell" was the Lisp REPL (never used one myself so I could be all wrong, of course).
Kids these days would of course use json instead, at least until the serious (tm) enterprise people would swoop in and replace it all with xml and ShellReplFactoryInstanceFactoryFactoryBean.
That ties it into the .Net ecosystem. Better to pass strictly data and not behavior but in a more structured way. Yet it should still be pretty simple so sth. like json would be the right direction to go.
Ruby is my goto language for replacing shell scripts and even the occasional one-liner. I find it much more suited to that task than Python (more shell-like features, better regex/text processing, more flexible program structure), and much better-designed than Perl.
It is about an interesting programming problem to do with text processing - involving finding word frequencies (apparently initially posed by Jon Bentley to Donald Knuth).
The above post (on my blog) has two solutions by me, in Python and shell, and links to the original discussion on another blog, where I first read about the problem - which was an interesting read, with many comments:
Inevitably when shell programming is mentioned, particularly bash, I hear quite a few devs/programmers talk about how "shell is bad and you shouldn't use it" maybe with an added "for anything not a one-off or over 20 lines"...
I completely disagree. As the sysadmin who has been the guy actually managing devs/programmers messes in prod, shell is the one thing that just works, is easily readable, is easily documented, is fairly easy to write, and generally keeps me from pulling my hair out or become violent with the devs.
Take it from the guy who actually has to wrangle all your strange python/perl/go/node... bullshit, shell is something you shouldn't be bashing.
It took a lot of pushing from a mentor for me to finally click with the overarching principles of shell and see how to master it and become productive with it. He really forced me into it as well, I wasn't a willing pupil for quite some time, but he kept persisting and finally I started listening to him. I'm so glad I did. I've never read any book or tutorial or guide that teaches what he taught me either, despite searching several times. Also, I don't think I could have learned to master shell if I hadn't learned functional programming first. The day I realised how amenable shell is to higher-order thinking was a real epiphany.
These days shell is a really important tool in my toolkit, and I can see just how much some of my colleagues are held back by not knowing it.
Anyway, I really enjoyed the video, it was a nice exposition.