Nice, I've been waiting for a good modern shell to replace ZSH. Fish just wasn't enough of an improvement for me to abandon ZSH. This is an area where most open-source developers have avoided, as it's a "solved" problem, but it really isn't.
I'm currently waiting on Elvish [1] to mature. It's written in Go [2], but they're developing a new language on top of it for shell scripting, which I tried for a week. It's a nice modernization and is incredibly fast but still very alpha.
You don't realize how slow ZSH/Bash is (even without plugins) until you try a modern shell, and there are very few good modern shells around.
Besides performance my primary complaint is just how awful coding in ZSH is. I've been doing it for years and I still feel like I'm running into walls constantly and struggling to write 'clean' code that isn't full of hacks and workarounds.
High performance + modern scripting language = my ideal future shell
> This is an area where most open-source developers have avoided, as it's a "solved" problem, but it really isn't.
I've found the complete opposite to be the case. This is one area where there are hundreds of different shells out there as it seems a "cool" thing to try since it's basically just an extension of writing your own language. Most never go beyond pet projects though.
If you're curious about Go shells, I'm writing one too[1]. It's also alpha but you can already write some reasonably complex scripts in it since it supports message passing, network sockets and background processes. It's definitely not fast though. Fast enough for what I need it to do but there's plenty of room for performance tuning once in beta.
> This is one area where there are hundreds of different shells out there... Most never go beyond pet projects though.
I have had the same experience. For every scripting/interpreted language out there, there is at least one person who thinks "ooh, wouldn't it be neat if I could use this as a shell?" or, almost as often, "to program my text editor?" Soon they realize that interactive programs like shells and text editors are hard to get right. There are thousands of special cases and non-obvious features that have only been found by hard experience. These programs must also interact with a computing world that has evolved over decades.
Most programmers give up at this point. A few persist, and of those, a few end up creating something genuinely useful to more than themselves and a couple of friends. I'm not sure if I would call shells a "solved" problem, but I don't believe the incredible amount of work required to make a sufficiently better one is worth it.
From an academic perspective I think it is totally worth the effort. It can teach you how to write a parser, a greater understanding of pseudo-TTYs, process IDs, and UNIX file streams (eg handling EOT bytes). Plus ANSI escape sequences (for readline support), globbing, etc. There's a lot of theory that can be learnt from writing even an unsuccessful shell. Even with myself who has been a Linux administrator for 10+ years and a developer for twice that, there were lessons I've learned when writing murex.
Program in rc shell[0]. It has a much superior syntax, and can manage piped commands. The use of the single quote to mean "really fucking quote", and $* working as every user would expect.. makes it a joy.
Secondly, I use rc in emacs shell-mode which poorly emulates a terminal, so advanced completion and all that other stuff I delegate to emacs. rc is just a programming language that supports shell commands extremely well.
I'm a big fan of fish, particularly with omf. Though honestly it has some performance issues, particularly if your prompt line calls something that could take some time to resolve (such as git status or a du).
I've been having big performance issues with tab completion. Also, I just don't find the language to be that much better than bash, and writing in fish means i can't share my shell scripts with any coworkers. I've been writing everything in bash lately even though i still use fish.
Same. I love fish as an interactive shell, but write all my scripts in Bash or some other, suitable scripting language. I don't mind having separate interactive and batch languages.
That's basically my biggest problem with fish. If I could get 99% of the functionality with near complete language incompatibility with bash, that would be ideal.
> Nice, I've been waiting for a good modern shell to replace ZSH.
I think xonsh is a good modern shell to replace bash/zsh/*sh. The only two issues are: 1) it has the stupidest name ever with an even more stupid pronunciation, 2) it's great, it's going toward a great direction, but it fails in one of your key areas for what makes a good shell: speed, it's very slow.
But regarding speed, please tell me how you find most shells to be too slow? Most of the time you're in shell you're not even doing shell stuff, you are rather using tools like ls and find and grep, etc. For the almost 2 decades I've been using bash, it never felt "slow" to me. xonsh feels slow, but at least the python-esque of it makes it a pleasure to use (except where you hit areas where it doesn't work well... I hope that improves over time though)
Sounds like what you want is more along the lines of the Ion shell[1], rather than Elvish. It's written in Rust, and the performance well exceeds Dash (written in C), despite having a lot of features that Dash is unable to do efficiently. Go's just not a good a language for writing a shell, especially once you get into job control and signal handling, or if you care about performance.
Some of the great features you won't find in Bash or Zsh are the string and array methods[2], first class arrays[3], slicing syntax[4], tuple assignments[5], optionally type-checked assignments[6], ability to use functions within pipelines, typed function parameters, a much simpler syntax, and more. Many of the best ideas from Fish, Oil, Elvish, Bash, Zsh and other shells have been implemented or are in the process of being implemented.
I've been using xonsh (shell + Python ) for about 18 months now to great effect. Bash stuff basically works as-is because of the operational semantics that completely separate shell stuff from Python stuff
If you're looking to contribute to something, the devs are really helpful and open to stuff (I was able to hack together an autopair feature pretty quickly and they took it)
I have to say I have been using BASH, SED and AWK less. They are great tools but in the end it is getting easier and easier to get things done with a Macro in Racket for me.
Looking back, I'd like to add Go to the list. (Not that I consider it hip, probably the least hip on the list. But well suited for writing a shell I guess.)
on a related note there's Upterm https://github.com/railsware/upterm which is a terminal emulator in node. I've found it pretty sucky for running fish shell in but i think it's ok for bash and zsh
Yeah, that's an old sentiment. I used to echo the same thing, but it's worth coming to terms with the fact that node is a pretty good platform to build on.
There are zero things I want my shell to do which depend on as much tooling as Node provides.
I want a compiled native binary with as few dependencies as possible for the shell I'm going to use in single-user mode with very few resources available.
JavaScript looks nothing like the “created in 10 days” version to which you speak. I would highly suggest you take an uncomfirmation(?) biased look at the language as it is now. Asyncs simplicity is arguably a reason why it’s a good platform to build off of. I agree I would rather have a single binary, and node may not be the best platform for shells on this point.
> JavaScript looks nothing like the “created in 10 days” version to which you speak.
It has improved, sure. But not only in the direction that makes me happy. But you cannot undo some bad design choices, without it becoming a different language.
> I would highly suggest [...]
I write JS at times. It hurts badly. Not everyone feels it. Maybe coming from PHP or Perl it does not even hurt that much. But after writing some Haskell and Elm lately, the pain got more intense.
> Asyncs simplicity is arguably a reason why it’s a good platform to build off of.
Not having choices is never a good thing in my book. And I do not think that JS does async particularly well. So much line noise when it comes to working with promises.
I don't know about PHP (my experience with it is limited), but coming from Perl, Perl is a breath of fresh air in comparison.
Then again, to me Haskell makes even the worst JAPH-style Perl one-liners look readable in comparison, so maybe I'm not the best judge of language design.
i'm with you @cies JS has had some of its edges smoothed out but it's not a great language and its answer to parallelism and concurrency is crap IMNSHO. And i'm someone who came from perl way back in the day.
> Though core.async makes async programming quite nice
Sure you can paint your cage gold, and it will look nicer. But it remains a cage. I want to choose, especially when it comes to how I do my concurrency.
Node may be OK for some things but it's not standard. I don't want to use a shell that I won't find anywhere but my own workstation. I want to use a shell that I will find on 99% of any systems I have to manage or work on. Even Python doesn't meet that bar and it's been around a lot longer than Node.
They worship Node because it gives them an earning opportunity while not having to put in any real effort in learning and thinking in my opinion. If C# was made mandatory for example I imagine this lets kode trend would die pretty quickly.
This is interesting, but I feel that making shell support such a first-class citizen is a mistake. Shell languages basically treat system binaries as functions, which causes you to lose a lot of power with everything being a string. Using system binaries (cat, echo, etc.) isn't really necessary for a modern command language due to the ubiquity of scripting languages and module systems. It's much more powerful to write procedures that work with structured data to perform similar actions.
One of the things that Shells have gotten right is postfix function composition (the | operator). This allows you to very easily compose functions to perform complex operations, and this isn't really present in scripting languages. Haskell let's you do this with the `&` operator, and you can write a procedure for Scheme that performs similarly. I'm not sure how this would work with Python or JS.
I'm hoping that in the next few years we'll see a larger break from shell syntax. The biggest thing I think holding this back is REPLs. Most scripting languages would do fine as a shell replacement with the addition of a stdlib for the purpose, but they need support for navigating the file system. Maybe tab-completion for paths like Bash does would be best, but I'm still thinking about it.
This is object-oriented method chaining. Not the same thing as functional composition (piping is essentially just left-to-right composition), which is what's mentioned in the OP.
It's also nowhere near as flexible, because it requires the return value to be responsible for providing a set of APIs for the next method call on every step of the chain, whereas each function in a functional composition can operate on any arbitrary input/output.
I think the concept of piping is actually more similar to function(function()) than .method().method(), because you're not limited to the scope of your initial object's methods.
And the truth is that functionB(functionA()) reads much worse than functionA | functionB , so yeah I agree that this is something that shell scripting languages got right.
For reference, various functional languages do indeed treat something like 'foo |> bar' as equivalent to 'bar(foo)' (I've used this extensively in Elixir, where it's standard practice; I'm pretty sure this is derived from F# and/or Clojure, both of which - IIRC - use the same operator for the same purpose).
It's not really perfect parity with what a shell typically offers, though, since a shell's pipeline steps (usually) run in parallel and indefinitely, consuming an input stream and producing an output stream (whether those streams are text, rich objects, audio/video data, or something else entirely). Elixir/Erlang are well-suited to achieving that parity by spinning up processes and using message passing for I/O, but Elixir's pipeline operator doesn't really do that (last I checked).
Exactly what yoavm said. The pipelining that shell languages do allows for more complex and more arbitrary data transformations. I feel that doing this in Python would be difficult, but I'm not very familiar with it. In scheme you can do something like this:
Wrote something similar in Python, even called it the same!
def pipe(init_val, *funcs):
"""Apply `init_val` to the composition of `funcs`
pipe(init, func1, func2, ..., funcn) -> funcn(...(func2(func1(init_val))))
"""
return functools.reduce(lambda val, func: func(val), funcs, init_val)
Check out clojure's threading macros[1] which extends that to taking forms, where the current value is spliced in as the first or last argument. e.g., with ->> (thread-last):
(->> '(3 4 5) (map 1+) (reduce +)) ;; 15
Fits the bill for short functions, and you can still have lambdas for when you really need them.
Author here, thanks for posting. The project is still very early in the development and there is a lot of work to be done to make it ready for daily use.
I would appreciate any feedback. What are the less known features of existing shells you really like? What things would you change about existing shells if you could?
You may also be interested in looking at https://www.oilshell.org/ . Like yours, it's an effort to make a 'better bash', and he has a lot of posts discussing various aspects of Bash syntax and shells in general, and how to improve them.
Yeah, Oil shell is great. I got a lot of inspiration and learn how things work by reading those posts. I try to tackle it by a different approch by leveraging Clojure syntax for the programming bits but keeping things compatible for the command execution uses.
The roadmap seems well thought out. Perhaps linking each unfinished item to a summary Github issue explaining how you see it being approached would assist with gaining contributors. You could then keep a list of "where to jump in" by assigning that sort of issue a tag ("start-hacking-here" or some such).
Awesome project, and absolutely well motivated. I would love a shell that improves on bash with sane syntax.
I'd suggest swapping stage 3 and stage 2 -- environment vars and signal and job control are more important (IMO) than everything in stage 2, and may affect the syntax more as well.
I think this kind of shell has the best features of something like PowerShell -- the ability to work with real data structures -- but combined with Clojure's opinionated minimalism about those data structures, the result seems more useful.
One other thing that Clojure has, however, which may be a problem here, is a strong emphasis on precise namespacing. That seems a like a serious problem for day to day use of this kind of shell. Pedantically namespacing all the interesting library functions I want to use would get old fast.
I really like this idea and I really like Clojure. Though I'm curious about performance. And do you know if anyone is doing something similar with Racket? (Looks like there is a Common Lisp shell http://www.cliki.net/CLISP-Shell)
eshell is another lispy shell that's been in Emacs for a very long time. Elisp is probably my least favorite lisp that's commonly available, but eshell probably has the largest user base and is certainly the most mature out of all of these contenders. You can reasonably use eshell as your primary shell.
I use eshell daily. It's nice to have a shell that also integrates with with list, but there's also a ton of value-add with integration with the emacs ecosystem.
For example in eshell you can do:
cd /ssh:some-server:/
top
cd /docker:container:/
find-file etc/hosts
Until you type '|', since eshell will buffer each process's output in an emacs buffer. It doesn't actually stream between pipes, which basically makes it useless for a complicated pipeline.
Thanks for the amazing work! This is a project that many would look forward too. Shell is a place where Clojure's program to abstractions and data oriented programming philosophy can do wonders.
Ability to browse documentation of Clojure functions at the shell might help. Currently, man pages can be accessed for utilities. But quickly browsing doc of, say, Clojure re-find will add value, although features from stage 2 and 3 of roadmap are more important/show-stoppers.
You can use node libraries by requiring them with (js/require). ClojureScript libraries are in theory possible too, but I need to figure out how to specify classpath options for them.
One thing that would be killer in a cljs shell is some sort of paredit capability. I've been writing clojure long enough with good tools that when I have to use it in an environment without paredit it slows me down a ton.
This looks great, though, and I can't wait to try it out.
Steady-state performance tends to be better on the JVM but startup performance is always much better on top of JavaScript. That makes it an enticing trade off for shells which is a process that you start new ones of constantly and where the performance of the shell itself is often not very important. (Performance-critical or hard-to-duplicate code like, say, line editing code, can be done equally well on either platform via FFI.)
Why not just make a server that starts up once in the background and then a lightweight c frontend to the server that essentially just pipes every keystroke to the server?
Shells are supposed to inherit a ton of state from their parent — [e]uid, groups, working dir, env vars, nice, umask, chroot, ... This takes zero effort in unix, but would all have to be (securely!) re-implemented to be able to send code to a shared execution daemon...
And I expect the right implementation for many of these params will be running separate daemons anyway (eg. you clearly want one per user).
Clojure/JVM apps are very slow to start, which makes them particularly unsuitable for a shell, where you typically don't want to wait a few seconds for the prompt to become interactive.
Then I just use figwheel to manage my cljs projects (I think this is the default now for reagent and luminus projects, it's also the default for re-natal for react native development in cljs). Life is pretty good and I have a nice debugger via Chrome.
I prefer the debugger in CIDER (which doesn't work for cljs) to the debugger in Chrome, but the Chrome debugger is decent (and it's what I'd have in Javascript anyway). There is a nice chrome extension to help with clojurescript eval in the browser/debugger called dirac: https://github.com/binaryage/dirac
This configuration is tremendously slow for me for some reason. Looks like Chrome put a tab "to sleep" when it loses focus. That leads to long freezes while you type anything in cljs file. Am I missing anything?
Whoa, awesome. That was super helpful. I'm also a spacemacs user, but my pain was mostly for non-browser Cljs. I should have specified, you're right: developing browser Cljs is pleasant because of figwheel :)
The downsides are pretty darn significant though. I can’t imagine a situation where I would trade ‘starts a bit faster’ for ‘throw away all parallelism in code’ personally... but hey. Plenty of people use node for very serious stuff.
I suppose I just think this is an example of fast and cheap on the fast/cheap/good scale.
But this is a shell. You start _all the time_, including plenty of times where you don't really notice: other processes often start subprocesses via your shell to get your environment right. Meanwhile, the shell is usually just waiting for you, so it's not like it's an intrinsically parallel process. This doesn't impact the shell's ability to run multiple things at once: the shell eventually creates subprocesses. One would expect an event-loop like Node to be perfect for this sort of thing.
It's a highly performant platform, based on a rapidly evolving language that's easy to understand. As such it's largely democratised server/shell programming to millions of users and web developers who could have been turned off by more advanced syntaxes and programming concepts. These people have created millions of open source packages which now dwarf most other communities, and the platform has unlocked an incredible amount of untapped economic value that powers likely hundreds of thousands of businesses around the world. Need I go on?
I don't think it is. In the words of Bill Clinton: "it's the economy, silly". Economic/social factors trump technical aspects 99% of the time, the 1% being technical revolutions. And Java or C# or Lisp being better than Javascript doesn't fall into the "revolution" category. Ergo Javascript everywhere :)
That’s answered in the comments elsewhere on this page. Mainly because shell processes are started frequently and the JVM start-up time would be much too slow.
The JVM is well-known to start slowly. It's not just clojure. For example, that's why people don't write command-line scripts in scala. I mean this is what I've always read, and is my experience from my limited contact with the JVM, and this is what people who use scala tell me. If you're going to contradict a widely-held belief then you need to provide some justification! Try googling "jvm slow startup".
Its slow because a bare bones hello world takes 40ms in Java instead of 1ms in C. Its "slow" relative to other languages, but its not really that slow over all. I can use mvn without noticing that its slow, but leiningen takes multiple seconds to run a command.
Note that the Clojure team are working on improving this, but it will never be as fast as pure Java and certainly never as fast as other languages (including cljs on node)
It seems like the concepts are orthogonal though. The reason powershell works is that it replaced the entire shell toolchain with object-oriented equivalents. Without that you're fundamentally limited because your inputs are still plain text, so you're still doing the equivalent of awk/sed scripts to transform text into structured data.
That aligns pretty much with my conclusion that to make an useful object shell, I'd be essentially rewriting the whole userland and probably some kernel bits too (/proc and /sys etc). So basically building a completely new OS on top of Linux kernel. I wouldn't expect such project to be very popular..
For quite a while now, I've fantasized about creating an object-oriented toy OS with an OO shell and something akin to Classic Mac OS's resource forks. As much information as possible would be stored as OO data structures in alternate file streams with native tooling to query them.
It would need to be based on Illumos and not Linux, though... IIRC, the only existing open-source filesystem that supports anything akin to a resource fork is ZFS, and the Linux kernel doesn't have the syscalls readily access it (ZFS implements extended attributes as forks, but the kernel has really tight limitations on what can be done with xattrs, much tighter than what ZFS can do).
It's just a fantasy, though, and not something I have the skills or the time to do.
Looking forward to the day I can write backend code in clojure, frontend code in clojurescript, write all my terminal commands in clojure and use an IDE that is configurable via clojure code.
One of the tricky things about shells is the interaction between fork and pthreads. For example, if you fork while another thread holds a lock, you risk deadlock.
node uses pthreads for certain asynchronous operations like stat(). How does Closh handle forking when there are multiple threads?
JS is single-threaded by design and node encourages use of asynchronous operations and then runs callbacks on the event loop. Node provides higher-level non-blocking function spawn, so closh currently does not need to do forking directly. I hope this can get us far before the need to drop into lower level and start dealing with threads.
I am curious, what does fish use threads for and how does it handle them?
Very cool! I use Clojure on a daily basis for coding, this is a great addition to the arsenal :)
Actually, my main argument against _not using it_ is that Bash has this awkward syntax that is hard to remember if you don't use it often...
But anyway, I keep forgetting it after years so I might as well embrace it and take the leap to use this.
Thanks for the initiative!
Scsh runs in a modified scheme48 repl - no problems with interactive use, though I've not run it as an actual login shell, though perportedly that works
I've been meaning to implement something like this for a while, but couldn't find the time to do it. It is nice to know that it's already done! I'm going to check it out.
ipython shell with its magic commands is pretty powerful, although not a bash replacement. For example, you can assign output of ls command to a variable (e.g. a = !ls) and then use that variable as a python list.
I'm currently waiting on Elvish [1] to mature. It's written in Go [2], but they're developing a new language on top of it for shell scripting, which I tried for a week. It's a nice modernization and is incredibly fast but still very alpha.
You don't realize how slow ZSH/Bash is (even without plugins) until you try a modern shell, and there are very few good modern shells around.
Besides performance my primary complaint is just how awful coding in ZSH is. I've been doing it for years and I still feel like I'm running into walls constantly and struggling to write 'clean' code that isn't full of hacks and workarounds.
High performance + modern scripting language = my ideal future shell
[1] https://elvish.io/
[2] https://github.com/elves/elvish