Here, exec associates the file descriptor (3 here, replace with any desired descriptor) with the pipe created by mkfifo. The filesystem path to the pipe is removed immediately after we obtain a file descriptor to it, so that the the only remaining reference to the pipe in the system would be from this script, and thus when the script dies, the kernel will automatically free the pipe.
Rephrased for clarity: POSIX specifies at least (a minimum of) 9. POSIX does not specify a maximum. Shells may provide more than 9. From the spec:
> Open files are represented by decimal numbers starting with zero. The largest possible value is implementation-defined; however, all implementations shall support at least 0 to 9, inclusive, for use by the application.
Despite writing shell scripts for nearly three decades, I also was unaware of what POSIX had to say but I can't recall ever needing more than a couple extra FDs at most.
"Modern Unix systems offer named pipes, also known as fifos, which can be used to hand-craft arbitrary process communication topologies. However, if combined one-to-many and many-to-one piping are setup by using named pipes, another problem will occur. Due to the limited buffering offered by typical programs, deadlocks can easily occur when a process consuming data from many producers with more than one input, blocks waiting for input from one of the processes feeding it. This can cause a second feeding process to block, waiting to send its output to another one of the consumer process’s inputs, and, thereby, blocking the upstream process feeding both processes that provide data to the consumer one."
dspinellis has commented on another dgsh discussion¹(along with you). Interestingly, with a light comparison to pipexec².
I stumbled upon pipexec trying to find a battle tested solution to extend a data munging task where I was relying on zsh's multios³, mostly because orchestrating the interactions with a coproc'd jq for output were fighting me. There is something both frustrating and soothing about finding a seven year old comment pointing out why my path was doomed before I'd even started; people have solved the problem already, plus people far smarter than me also found the trap.
We could do something like dgsh, but so far I haven't seen a lot of uptake / demand. Every time it's mentioned, somebody kinda wants it, and then it kinda peters out again ... still possible though.
I think flat files work fine for a lot of use cases, and once you add streaming, you also want monitoring, more control over backpressure/queue sizes, etc.
I use mutlios and even I'm not that attached to it. The majority of my use is combined with process substitution, and could be replaced with common-ish tools like pee¹(or pipexec for more complex cases). The only occasion when I'm thankful for it is if I want to use a shell function as a target, but there are workarounds for that too.
As a noclobber user the footgun is largely hidden to me, but I feel its presence. multios without globbing support would be less useful, but would still work for most of my use cases. Scanning my shell history I see various cases of relying on zsh's ability to apply sorting and filtering to globs with multios' input redirection, but only a couple where I want that in output redirection. The input instances could easily be rewritten using cat and globbing.
Even with multios unset the behaviour is different between zsh and bash. For example, nomultios disables all the expansion, so zsh behaves like more like dash with ': >t{1,2}' creating a file instead producing an error like bash does.
[FWIW, I google'd multiios to link the option in mt original comment. It really feels like it needs double-i, and I read the single i name the same way you do.]
---
I'd be one of those people whose desire for dgsh-like functionality wanes. If it was slight DSL that I could "upgrade" pipelines to I'd probably use it, but not enough to warrant working on it or switching other tooling to support it.
The end of result of this morning's pipeline was breaking my jobs up, and applying some judicious use of nq² to keep track of it. I'd follow your advice and move on to more specialist tools if the job grew significantly or if it became a regular occurrence.
This is neat, but outside of a contrived ouroboros example, what’s a real world use case for this?
There’s a natural flow of outputs becoming inputs and I’m struggling to identify a situation where I would feed things back into the source. Also, named pipes kind of solve that already.
Agreed -- their only example is `pipexec -l 2 -- [ LS /bin/ls -l ] [ GREP /bin/grep LIC ] '{LS:1>GREP:0}'` which appears to be `ls | grep LIC` with more steps. Seems like a (cool) solution without a real problem.
(I'd love to be wrong though and see a real use case for some cool feedback loop of commands)
I recently wanted this for some scripting over SSH. I basically want to run a script on a remote machine and read it back, but implement it as a function instead of a wrapper around SSH.
Chatbots, where the bot only needs to be line-driven and you can connect it to any CLI chat interface. Or perhaps, run your AI agent attached to a shell, and have it treat standard IO as a shell session.
I suppose such feedback could be used for reaching a fixpoint. Suppose you have a build system that reads targets to be built from stdin and outputs to stdout targets that are dependent on that target and must now be rebuilt. With an ouroboros, the build system will continue to run, even if the dependency graph is dynamically cyclical, until the fixpoint is reached and the build terminates.
It can be used as client/server communication locally. I’ve done that (without pipexec) for bots in a multiplayer game. That way I could implement the bot ai independently and test them against each other.
I did pretty much this exact same thing too. The game runner spawns 2 bots and acts as the middleman/, piping their output to each other. I did it in node though. I wouldn't want to code this in bash.
This reminds me of MIT's open courseware, 601 SC, unit 1 with state machines, going all the way to build Fibonacci with them without recursion, tbh the moment the teacher translated the state machine to bounds to electrical circuit I felt it was a leap and I couldn't quite understand their relationship, maybe I missed a requirement course. I tried to express that course in Rust as one of my first projects learning the language here https://github.com/sebosp/rustexercises/blob/develop/ocw601s... and I think a similar iteration in the direction of this project would be to build the dependencies as drag-and-drop boxes over the browser (maybe egui) and connect the state machines by clicks, maybe download the generated code as either bash or compilable rust code, you know, for kids.
What a beautifully designed tool. In our Python codebase we end up reaching for inline sh scripting a lot whenever we need to pipe between processes. In a way it feels ok — after all, no one has any qualms about reaching for inline SQL to get things done, so what’s wrong with a little shell script in the middle of a Python module?
Just as there are efforts — both wise and misguided — to represent the building of an SQL query with Python syntax, what Python tools are there to build sh pipelines between processes with a more pythonic syntax? Do they provide value in excess of the novelty tax one has to pay for using a non standard library?
It's an interesting package, and I have some of the use cases it appears to address. However, the documentation is inadequate to quickly understand how to robustly build some of the more complex cases. In particular, how to build bash-style process substitution. Robust here is the pipeline exits non-zero if any of the substituted processes fail, as demonstrated by this example:
#!/bin/bash
set -beEu -o pipefail
cat <(date) <(false)
echo did not exit non-zero
If this is addressed, it would be worth more time to figure out Pipexec.
Considering how airflow/dagster are trendy these days, concurrency too.. I assume a leaner, os/language agnostic solution for this problem might emerge not too far in the future.
It would be cool to have a tool similar to this, but for composing a graph of commands similar to aws step functions (or Jenkins pipelines). I’d call it mapexec :)
~ $ man stdin | grep stdin,
stdin, stdout, stderr - standard I/O streams
The input stream is referred to as "standard input"; the output stream is referred to as "standard output"; and the error stream
is referred to as "standard error". These terms are abbreviated to form the symbols used to refer to these files, namely stdin, stdout, and stderr.
On program startup, the integer file descriptors associated with the streams stdin, stdout, and stderr are 0, 1, and 2, respectively.
This alignment is, indeed, very much deliberate. Take a peek at this:
No need to pick sides. Manuals are a best effort by people writing code to communicate their intent. RTFM is an expression of frustration that people don't pay attention to their efforts before spouting off.
An example use case would be like so: https://unix.stackexchange.com/a/216475/585293