A few notes from a recent attempt to use subprocess in the middle of a few gener...

A few notes from a recent attempt to use subprocess in the middle of a few generators:

p.communicate() only works if the output of the program can fit in memory. If it can't, python will gladly crash your box for you.

The buffer size of the os pipes on Linux are 64k. If you don't do your reads and writes in this size, it's possible to deadlock yourself.

Reads on the pipes has to be done in a non-blocking fashion, using os.read(), if you're dealing with streams of data that are larger than the memory you have available to yourself.

You can chain multiple subprocesses together directly, if you feed the stdout pipe to stdin on the next process.

    p1 = subprocess.Popen(['cat', 'foo'], stdout=subprocess.PIPE)
    p2 = subprocess.Popen(['grep', 'cat'], stdin=p1.stdout, stdout=subprocess.PIPE)
    p2.communicate()

select.poll() works on the os.pipes which are contained in the Popen objects.