Named Pipes in Bash

[nkurz]

http://www.hollenback.net/NamedPipesInBash

[pstadler]

You can create named pipes explicitly:

    $ mkfifo ~/pipe
    $ ./background.sh > ~/pipe &
    $ tail -f ~/pipe

http://www.linuxjournal.com/article/2156

[runn1ng → pstadler]

Yes, but I have always found that cumbersome and I was never sure which program has which pipe, and what will exactly happen if one of the reading/writing programs dies. (I am still not sure.)

So I am glad that what OP wrote exists.

[ams6110 → runn1ng]

One thing to keep in mind is that if nothing ever consumes the data in the pipe, the pipe stays around and you eventually will run out of open file descriptors. So if you're scripting with named pipes, be sure you have adequate cleanup in case of errors.

[recuter → ams6110]

Am I right in guessing there will be approx. 65536 file descriptors available? :)

[mtdewcmu → runn1ng]

True. I've known about that feature of bash, but I have found precious few good opportunities to use it. As someone mentioned below, if a program expects a file and you give it a pipe, it may try to seek and cause the whole thing to fail. So trying to use it is always a gamble. Moreover, a named pipe is an object in the filesystem that bash has to create, only to remove it a moment later. So it is not really any cleaner than using a temp file. The only case where you'd clearly benefit is when the amount of data is too big to justify temporarily writing it out to a file.

[thwarted → mtdewcmu]

bash doesn't create named pipe filesystem objects when you use process substitution, it passes file descriptors via the dynamically created kernel managed /dev/fd paths. There is nothing to clean up.

[mtdewcmu → thwarted]

Oh, I think I see what you're saying. So calling them named pipes is inaccurate, then. If I'm not mistaken, using features of the /dev filesystem would be dependent on Linux, then. Either way, it isn't really a very robust design. Not like a pipe returned by pipe(2) and passed to a program that is expecting it. Simple unnamed pipes are a fundamental part of unix.

Here's what the man page says:

       Process  substitution  is  supported on systems that support named pipes (FIFOs) or the /dev/fd method of
       naming open files.  It takes the form of <(list) or >(list).  The process list is run with its  input  or
       output  connected  to  a FIFO or some file in /dev/fd.  The name of this file is passed as an argument to
       the current command as the result of the expansion.  If the >(list) form is used,  writing  to  the  file
       will  provide input for list.  If the <(list) form is used, the file passed as an argument should be read
       to obtain the output of list.

So it's done with /dev/fd if that's available, which basically means Linux with that filesystem mounted. Otherwise it falls back on named pipes.

[saljam]

There's a caveat with process substitution; the command can't seek() on the input file.

If if a command, for example, needs to do 2 passes, it'll will either have to be 'smart' enough to stick the output in a buffer or fail.

[eatitraw → saljam]

Yes, this is a problem in bash. In zsh you can use temporary files for process substitution, like this:

vimdiff =(ls /bin) =(ls /usr/bin)

You can use it even from bash:

zsh -c 'vimdiff =(ls /bin) =(ls /usr/bin)'

[e12e → eatitraw]

Actually vimdiff seems to have no problem with:

    vimdiff <(ls /bin) <(ls /usr/bin)

[dllthomas]

This and so much more is in http://www.tldp.org/LDP/abs/html - well worth perusing if you're needing to do a lot of bash.

[recuter → dllthomas]

Future readers, #bash on freenode says of this link:

< greybot> The infamous "Advanced" Bash Scripting Guide should be avoided unless you know how to filter out the junk. It will teach you to write bugs, not scripts. In that light, the BashGuide was written: http://mywiki.wooledge.org/BashGuide

They felt strongly enough to learn their bot of it. Keep this in mind, I haven't read much of either resource yet.

[dllthomas → recuter]

Interesting. I'd not heard anything negative about it, and found it useful both in my day-to-day BASH-ing and in a position where I maintained a huge pile of BASH scripts. It may very well be that I knew how to filter out the junk. There is certainly a lot that doesn't belong in scripts but is quite useful interactively.

[perlgeek]

Are these really named pipes? And if yes, what are the names? Kinda looks like anonymous pipes to my untrained eyes.

[gnaritas → perlgeek]

They're anonymous named pipes. Do this in a shell to see the name:

    echo <(true)

[Pitarou → gnaritas]

anonymous named?

We really need to refactor our terminology. :)

[gnaritas → Pitarou]

Why, that's exactly what it is? It's a named pipe, but not one used explicitly by you but implicitly by the process substitution feature to pass a file, which of course has a name, to another process. It's only anonymous to you because you didn't name it, to the system it's an ordinary named pipe.

[hamburglar → Pitarou]

Anonymous in that their name doesn't matter and you need not care about it. They are implemented using pipes with actual names, but the names aren't important from the shell user's point of view.

[vsbuffalo]

I wrote a post about this a while back, which may be of some help to some. http://vincebuffalo.com/2013/08/08/the-mighty-named-pipe.htm...

[nkurz → vsbuffalo]

Great article, thanks! You should submit it separately if you haven't. Noticed a typo in the title that you might fix first: "Using Names Pipes"

[taway2012]

Pretty cool.

Related: anybody has ideas to achieve the following?

display the stdout of a program on the console and also write a gzipped version of stdout to a file.

I currently jump through some hoops (involving two term windows) to achieve. 'tee -a' to a named pipe in one window and in another window, 'gzip <' from that named pipe. Would prefer a single-window, single-command solution.

[ColinWright → taway2012]

What's wrong with:

    cmd | tee >( gzip - > cmd.out.gz )

[taway2012 → ColinWright]

Yeah, I think this is probably going to work. Thx!

PS: 'inevitable', you made the same suggestion. But you seem to be hellbanned.

[runejuhl → taway2012]

Something like this?

    ls -al | tee >(cat) | gzip -c

[runejuhl → runejuhl]

...or you could run something like

    cat /proc/`pidof program`/fd/2

That, of course, won't eliminate the need for a second terminal.

(change to your needs, stdin==1, stdout==2, stderr=3)

[_paulc → taway2012]

ls -al | gzip -c | tee out.gz | gzip -dc

[zokier]

Very useful, hopefully I remember this when I need it next time (that being the bane of all bash cleverness). The >(foo) would benefit from some extra explanation though, currently it is not very clear how it differs from regular |foo.

[bartbes → zokier]

>(foo) turns into a file name, so it can be passed to things that absolutely want a file name to write to. And multiple occurrences of this should work as you'd expect.

[e12e → bartbes]

Interesting, indeed it does:

    # define a function that echoes it's first argument:
    $ arg1() { echo $1 ; }

    $ arg1 test
    test

    $ arg1 <(echo hello)
    /dev/fd/63

[Hello71 → e12e]

  echo <(:)

[e12e → Hello71]

Thanks, I've never investigated ":" beyond noting it's presence in fork bombs.

    :
    Null command.
    
    No effect; the command does nothing.
    
    Exit Status:
    Always succeeds.

[taejo → e12e]

Basically a builtin for `true`. Its use in forkbombs is unrelated except for the fact that it's a valid character in a bash function name. Defining `:(){:|:&)` obviously shadows the builtin.

[kps]

This 'bash' feature, among others, was in ksh before bash existed.

[panzi]

If I would have seen this one or two hours ago it could have spared me hate on irc://irc.freenode.net/bash