Programming Python (32 page)

The two processes of
the prior section’s example engage in a simple dialog,
but it’s already enough to illustrate some of the dangers lurking in
cross-program communications. First of all, notice that both programs
need to write to
stderr
to display
a message; their
stdout
streams are
tied to the other program’s input stream. Because processes share file
descriptors,
stderr
is
the same in both parent and child, so status messages
show up in the same place.

More subtly, note that both parent and child call
sys.stdout.flush
after they print text to
the output stream. Input requests on pipes normally block the caller
if no data is available, but it seems that this shouldn’t be a problem
in our example because there are as many writes as there are reads on
the other side of the pipe. By default, though,
sys.stdout
is
buffered
in this context, so the
printed text may not actually be transmitted until some time in the
future (when the output buffers fill up). In fact, if the flush calls
are not made, both processes may get stuck on some platforms waiting
for input from the other—input that is sitting in a buffer and is
never flushed out over the pipe. They wind up in a
deadlock
state, both blocked on
input
calls waiting for events that never
occur.

Technically, by default
stdout
is just
line-buffered
when connected to a terminal, but
it is
fully buffered
when connected to other
devices such as files, sockets, and the pipes used here. This is why
you see a script’s printed text in a shell window immediately as it is
produced, but not until the process exits or its buffer fills when its
output stream is connected to something else.

This output buffering is really a function of the system
libraries used to access pipes, not of the pipes themselves (pipes do
queue up output data, but they never hide it from readers!). In fact,
it appears to occur in this example only because we copy the pipe’s
information over to
sys.stdout
, a
built-in file object that uses stream buffering by default. However,
such anomalies can also occur when using other cross-process
tools.

In general terms, if your programs engage in a two-way dialog
like this, there are a variety of ways to avoid buffering-related
deadlock problems:

Thread can avoid blocking a main GUI, too, but really just
delegate the problem (the spawned thread will still be deadlocked). Of
the options listed, the first two—manual flushes and command-line
arguments—are often the simplest solutions. In fact, because it is so
useful, the second technique listed above merits a few more words. Try
this: comment-out all the
sys.stdout.flush
calls in Examples
5-22
and
5-23
(the files
pipes.py
and
pipes-testchild.py
) and change the parent’s spawn
call in
pipes.py
to this (i.e., add a
-u
command-line argument):

spawn('python', '-u', 'pipes-testchild.py', 'spam')

Then start the program with a command line like this:
python -u pipes.py
. It will work as it did
with the manual
stdout
flush calls,
because
stdout
will be operating in
unbuffered mode in both parent and child.

We’ll revisit the effects of unbuffered output streams in
Chapter 10
, where we’ll code a simple GUI that
displays the output of a non-GUI program by reading it over both a
nonblocking socket and a pipe in a thread. We’ll explore the topic
again in more depth in
Chapter 12
, where we
will redirect standard streams to sockets in more general ways.
Deadlock in general, though, is a bigger problem than we have space to
address fully here. On the other hand, if you know enough that you
want to do IPC in Python, you’re probably already a veteran of the
deadlock wars.

Anonymous pipes allow related tasks to communicate but are not
directly suited for independently launched programs. To allow the
latter group to converse, we need to move on to the next section and
explore devices that have broader
visibility.

In Python, named
pipe files are created with the
os.mkfifo
call,
which is available today on Unix-like platforms, including Cygwin’s
Python on Windows, but is not currently available in standard Windows
Python. This call creates only the external file, though; to send and
receive data through a fifo, it must be opened and processed as if it
were a standard file.

To illustrate,
Example 5-24
is a derivation of the
pipe2.py
script listed in
Example 5-20
, but rewritten here to
use fifos rather than anonymous pipes. Much like
pipe2.py
, this script opens the fifo using
os.open
in the child for low-level
byte string access, but with the
open
built-in in the parent to treat the
pipe as text; in general, either end may use either technique to treat
the pipe’s data as bytes or text.

"""
named pipes; os.mkfifo is not available on Windows (without Cygwin);
there is no reason to fork here, since fifo file pipes are external
to processes--shared fds in parent/child processes are irrelevent;
"""
import os, time, sys
fifoname = '/tmp/pipefifo'                       # must open same name
def child():
pipeout = os.open(fifoname, os.O_WRONLY)     # open fifo pipe file as fd
zzz = 0
while True:
time.sleep(zzz)
msg = ('Spam %03d\n' % zzz).encode()     # binary as opened here
os.write(pipeout, msg)
zzz = (zzz+1) % 5
def parent():
pipein = open(fifoname, 'r')                 # open fifo as text file object
while True:
line = pipein.readline()[:-1]            # blocks until data sent
print('Parent %d got "%s" at %s' % (os.getpid(), line, time.time()))
if __name__ == '__main__':
if not os.path.exists(fifoname):
os.mkfifo(fifoname)                      # create a named pipe file
if len(sys.argv) == 1:
parent()                                 # run as parent if no args
else:                                        # else run as child process
child()

Because the fifo exists independently of both parent and child,
there’s no reason to fork here. The child may be started independently
of the parent as long as it opens a fifo file by the same name. Here,
for instance, on Cygwin the parent is started in one shell window and
then the child is started in another. Messages start appearing in the
parent window only after the child is started and begins writing
messages onto the
fifo file:

[C:\...\PP4E\System\Processes] $
python pipefifo.py
# parent window
Parent 8324 got "Spam 000" at 1268003696.07
Parent 8324 got "Spam 001" at 1268003697.06
Parent 8324 got "Spam 002" at 1268003699.07
Parent 8324 got "Spam 003" at 1268003702.08
Parent 8324 got "Spam 004" at 1268003706.09
Parent 8324 got "Spam 000" at 1268003706.09
Parent 8324 got "Spam 001" at 1268003707.11
Parent 8324 got "Spam 002" at 1268003709.12
Parent 8324 got "Spam 003" at 1268003712.13
Parent 8324 got "Spam 004" at 1268003716.14
Parent 8324 got "Spam 000" at 1268003716.14
Parent 8324 got "Spam 001" at 1268003717.15
...etc: Ctrl-C to exit...
[C:\...\PP4E\System\Processes]$
file /tmp/pipefifo
# child window
/tmp/pipefifo: fifo (named pipe)
[C:\...\PP4E\System\Processes]$
python pipefifo.py -child
...Ctrl-C to exit...

By mapping
communication points to a file system entity accessible
to all programs run on a machine, fifos can address a broad range of
IPC goals on platforms where they are supported. For instance,
although this section’s example runs independent programs, named pipes
can also be used as an IPC device by both in-process threads and
directly forked related processes, much as we saw for anonymous pipes
earlier.

By also supporting unrelated programs, though, fifo files are
more widely applicable to general client/server models. For example,
named pipes can make the GUI and
command
-line debugger integration I
described earlier for anonymous pipes even more flexible—by using fifo
files to connect the GUI to the non-GUI debugger’s streams, the GUI
could be started independently when needed.

Sockets provide similar functionality but also buy us both
inherent network awareness and broader portability to Windows—as the
next section
explains.