Programming Python (109 page)

To make some of this more concrete,
Example 12-12
illustrates how some
of these complexities apply to redirected standard streams, by
attempting to connect them to both text and binary mode files produced
by
open
and accessing them with
print
and
input
built-ins much as redirected script
might.

"""
test effect of connecting standard streams to text and binary mode files
same holds true for socket.makefile: print requires text mode, but text
mode precludes unbuffered mode -- use -u or sys.stdout.flush() calls
"""
import sys
def reader(F):
tmp, sys.stdin = sys.stdin, F
line = input()
print(line)
sys.stdin = tmp
reader( open('test-stream-modes.py') )         # works: input() returns text
reader( open('test-stream-modes.py', 'rb') )   # works: but input() returns bytes
def writer(F):
tmp, sys.stdout = sys.stdout, F
print(99, 'spam')
sys.stdout = tmp
writer( open('temp', 'w') )             # works: print() passes text str to .write()
print(open('temp').read())
writer( open('temp', 'wb') )            # FAILS on print: binary mode requires bytes
writer( open('temp', 'w', 0) )          # FAILS on open: text must be unbuffered

When run, the last two lines in this script both fail—the second
to last fails because
print
passes
text strings to a binary-mode file (never allowed for files in
general), and the last fails because we cannot open text-mode files in
unbuffered mode in Python 3.X (text mode implies Unicode encodings).
Here are the errors we get when this script is run: the first run uses
the script as shown, and the second shows what happens if the second
to last line is commented out (I edited the exception text slightly
for
presentation
):

C:\...\PP4E\Internet\Sockets>
test-stream-modes.py
"""
b'"""\r'
99 spam
Traceback (most recent call last):
File "C:\...\PP4E\Internet\Sockets\test-stream-modes.py", line 26, in 
writer( open('temp', 'wb') )            # FAILS on print: binary mode...
File "C:\...\PP4E\Internet\Sockets\test-stream-modes.py", line 20, in writer
print(99, 'spam')
TypeError: must be bytes or buffer, not str
C:\...\PP4E\Internet\Sockets>
test-streams-binary.py
"""
b'"""\r'
99 spam
Traceback (most recent call last):
File "C:\...\PP4E\Internet\Sockets\test-stream-modes.py", line 27, in 
writer( open('temp', 'w', 0) )          # FAILS on open: text must be...
ValueError: can't have unbuffered text I/O

The same rules apply to socket wrapper file objects created with
a socket’s
makefile
method—they
must be opened in text mode for
print
and should be opened in text mode for
input
if we wish to receive text
strings, but text mode prevents us from using fully unbuffered file
mode altogether:

>>>
from socket import *
>>>
s = socket()
# defaults to tcp/ip (AF_INET, SOCK_STREAM)
>>>
s.makefile('w', 0)
# this used to work in Python 2.X
Traceback (most recent call last):
File "C:\Python31\lib\socket.py", line 151, in makefile
ValueError: unbuffered streams must be binary

Text-mode socket wrappers
also accept a buffering-mode argument of
1
to specify
line-buffering
instead of the default full
buffering:

>>>
from socket import *
>>>
s = socket()
>>>
f = s.makefile('w', 1)
# same as buffering=1, but acts as fully buffered!

This appears to be no different than full buffering, and still
requires the resulting file to be flushed manually to transfer lines
as they are produced. Consider the simple socket server and client
scripts in Examples
12-13
and
12-14
. The server simply reads three
messages using the raw socket interface.

from socket import *           # read three messages over a raw socket
sock = socket()
sock.bind(('', 60000))
sock.listen(5)
print('accepting...')
conn, id = sock.accept()       # blocks till client connect
for i in range(3):
print('receiving...')
msg = conn.recv(1024)      # blocks till data received
print(msg)                 # gets all print lines at once unless flushed

The client in
Example 12-14
sends three
messages; the first two over a socket wrapper file, and the last using
the raw socket; the manual flush calls in this are commented out but
retained so you can experiment with turning them on, and sleep calls
make the server wait for data.

import time                            # send three msgs over wrapped and raw socket
from socket import *
sock = socket()                        # default=AF_INET, SOCK_STREAM (tcp/ip)
sock.connect(('localhost', 60000))
file = sock.makefile('w', buffering=1) # default=full buff, 0=error, 1 not linebuff!
print('sending data1')
file.write('spam\n')
time.sleep(5)               # must follow with flush() to truly send now
#file.flush()               # uncomment flush lines to see the difference
print('sending data2')
print('eggs', file=file)    # adding more file prints does not flush buffer either
time.sleep(5)
#file.flush()               # output appears at server recv only upon flush or exit
print('sending data3')
sock.send(b'ham\n')         # low-level byte string interface sends immediately
time.sleep(5)               # received first if don't flush other two!

Run the server in one window first and the client in another (or
run the server first in the background in Unix-like platforms). The
output in the server window follows—the messages sent with the socket
wrapper are deferred until program exit, but the raw socket call
transfers data immediately:

C:\...\PP4E\Internet\Sockets>
socket-unbuff-server.py
accepting...
receiving...
b'ham\n'
receiving...
b'spam\r\neggs\r\n'
receiving...
b''

The client window simply displays “sending” lines 5 seconds
apart; its third message appears at the server in 10 seconds, but the
first and second messages it sends using the wrapper file are deferred
until exit (for 15 seconds) because the socket wrapper is still fully
buffered. If the manual flush calls in the client are uncommented,
each of the three sent messages is delivered in serial, 5 seconds
apart (the third appears immediately after the second):

C:\...\PP4E\Internet\Sockets>
socket-unbuff-server.py
accepting...
receiving...
b'spam\r\n'
receiving...
b'eggs\r\n'
receiving...
b'ham\n'

In other words, even when line buffering is requested, socket
wrapper file writes (and by association, prints) are buffered until
the program exits, manual flushes are requested, or the buffer becomes
full.

The short story here is this: to avoid delayed outputs or
deadlock, scripts that might send data to waiting programs by printing
to wrapped sockets (or for that matter, by using
print
or
sys.stdout.write
in general) should do one
of the following:

The latter option may be more direct (and the redirection
utility module also returns the raw socket in support of such usage),
but it isn’t viable in all scenarios, especially for existing or
multimode scripts. In many cases, it may be most straightforward to
use manual flush calls in shell-oriented programs whose streams might
be linked to other programs through sockets.

Also keep in mind that
buffered streams and deadlock are general issues that go
beyond socket wrapper files. We explored this topic in
Chapter 5
; as a quick review, the nonsocket
Example 12-15
does not fully
buffer its output when it is connected to a terminal (output is only
line buffered when run from a shell command prompt), but does if
connected to something else (including a socket or pipe).

# output line buffered (unbuffered) if stdout is a terminal, buffered by default for
# other devices: use -u or sys.stdout.flush() to avoid delayed output on pipe/socket
import time, sys
for i in range(5):
print(time.asctime())                 # print transfers per stream buffering
sys.stdout.write('spam\n')            # ditto for direct stream file access
time.sleep(2)                         # unles sys.stdout reset to other file

Although text-mode files are required for Python 3.X’s
print
in general, the
-u
flag still works in 3.X to suppress full
output stream buffering. In
Example 12-16
, using this flag makes
the spawned script’s printed output appear every 2 seconds, as it is
produced. Not using this flag defers all output for 10 seconds, until
the spawned script exits, unless the spawned script calls
sys.stdout
.
flush
on each iteration.

# no output for 10 seconds unless Python -u flag used or sys.stdout.flush()
# but writer's output appears here every 2 seconds when either option is used
import os
for line in os.popen('python -u pipe-unbuff-writer.py'):    # iterator reads lines
print(line, end='')                                     # blocks without -u!

Following is the reader script’s output; unlike the socket
examples, it spawns the writer automatically, so we don’t need
separate windows to test. Recall from
Chapter 5
that
os.popen
also accepts a buffering argument
much like
socket.makefile
, but it
does not apply to the spawned program’s stream, and so would not
prevent output buffering in this case.

C:\...\PP4E\Internet\Sockets>
pipe-unbuff-reader.py
Wed Apr 07 09:32:28 2010
spam
Wed Apr 07 09:32:30 2010
spam
Wed Apr 07 09:32:32 2010
spam
Wed Apr 07 09:32:34 2010
spam
Wed Apr 07 09:32:36 2010
spam

The net effect is that
-u
still works around the steam buffering issue for connected programs in
3.X, as long as you don’t reset the streams to other objects in the
spawned program as we did for socket redirection in
Example 12-11
. For socket
redirections, manual flush calls or replacement socket wrappers may be
required.

So why use
sockets in this redirection role at all? In short, for
server independence and networked use cases. Notice how for command
pipes it’s not clear who should be called “server” and “client,” since
neither script runs perpetually. In fact, this is one of the major
downsides of using command pipes like this instead of sockets—because
the programs require a direct
spawning
relationship
, command pipes do not support longer-lived or
remotely running servers the way that sockets do.

With sockets, we can start client and server independently, and
the server may continue running perpetually to serve multiple clients
(albeit with some changes to our utility module’s listener
initialization code). Moreover, passing in remote machine names to our
socket redirection tools would allow a client to connect to a server
running on a completely different machine. As we learned in
Chapter 5
, named pipes (fifos) accessed with
the
open
call support stronger
independence of client and server, too, but unlike sockets, they are
usually limited to the local machine, and are not supported on all
platforms.

Experiment with this code on your own for more insight. Also try
changing
Example 12-11
to run
the client function in a spawned process instead of or in addition to
the server, with and without flush calls and
time.sleep
calls to defer exits; the
spawning structure might have some impact on the soundness of a given
socket dialog structure as well, which we’ll finesse here in the
interest of space.

Despite the care that must be taken with text encodings and
stream buffering, the utility provided by
Example 12-10
is still arguably
impressive—prints and input calls are routed over network or
local-machine socket connections in a largely automatic fashion, and
with minimal changes to the nonsocket code that uses the module. In
many cases, the technique can extend a script’s applicability.

In the next section, we’ll use the
makefile
method again to wrap the socket in
a file-like object, so that it can be read by lines using normal
text-file method calls and techniques. This isn’t strictly required in
the example—we could read lines as byte strings with the socket
recv
call, too. In general, though,
the
makefile
method comes in handy
any time you wish to treat sockets as though they were simple files.
To see this at work, let’s
move on.