PaulSD.com
Using a FIFO as a Detachable Pipe
(Created: 08/19/2010)


Anyone who knows anything about Linux/UNIX/BSD knows about pipes (command1 | command2), which cause two commands to be run simultaneously, with the output of one dumped into the input of the other. But have you ever needed a detachable pipe, to allow either the writer or reader command to be replaced with another command on the fly without killing the associated reader or writer?

Admittedly, this is an extremely rare need, since there is almost always another way to solve the problem without doing anything special. For example:

To dump the sequential output of multiple writers into a reader:
(command1 ; command2) | command3

To manually add additional output after a writer has completed:
(command1 ; cat) | command2

To dump each line of output into a separate reader:
command1 | xargs -I % "echo \"%\" | command2"

To start the writer and reader independently:
(command1 > data &) ; tail -f data | command2

To start the writer and reader independently without storing the data in an intermediate file:
mkfifo fifo ; (command1 > fifo &) ; command2 < fifo

To dump the output of multiple independent writers sequentially into a reader:
touch data ; (tail -f data | command3 &) ; command1 >> data ; command2 >> data

To dump the output of a writer into several readers in parallel:
mkfifo fifo ; (command3 < fifo &) ; command1 | tee fifo | command2

However, on a few occasions, I have needed to dump the output of a writer into multiple independently-started readers sequentially (but tail can't tell you where the last reader left off in an intermediate file, and "command > fifo" will quit if the FIFO's reader quits), or I have needed to dump the output of multiple independently-started writers sequentially into a reader for a very large data set or over a very long time period (so an intermediate file is impractical, and "command < fifo" will quit if the FIFO's writer quits). As far as I know, there is no trivial way to do this.

So, I've written a few simple programs which write to or read from a FIFO without quitting when the reader/writer quits: fifo_write.c fifo_read.c fifo_head.c (compile with "gcc -o /usr/local/bin/fifo_X fifo_X.c")

To dump the output of a writer into multiple independently-started readers sequentially:
mkfifo fifo ; (command1 | fifo_write fifo &) ; command2 < fifo ; command3 < fifo

To dump the output of multiple independently-started writers sequentially into a reader:
mkfifo fifo ; (fifo_read fifo | command3 &) ; command1 > fifo ; command2 > fifo

Of course, fifo_write and fifo_read can be used together if both multiple writers and multiple readers are needed:
mkfifo fifo ; ((command1 | fifo_write fifo ; command2 | fifo_write fifo) &) ; ((fifo_read fifo | command3 ; fifo_read fifo | command4) &)

fifo_head may be used to read a single line from a FIFO, then quit. I came across one particular problem where I needed to read one line from the FIFO at a time, but the obvious solution (head -n 1 fifo | command) reads and discards all of the data from the FIFO, then prints the first line, which is not what I wanted. Example usage:
mkfifo fifo ; (command1 | fifo_write fifo &) ; fifo_head fifo | command2 ; fifo_head fifo | comand3