Programming Python (19 page)

Earlier in this section,
we piped
teststreams.py
output into
the standard
more
command-line
program with a command like this:

C:\...\PP4E\System\Streams>
python teststreams.py < input.txt | more

But since we already wrote our own “more” paging utility in Python
in the preceding chapter, why not set it up to accept input from
stdin
too? For example, if we change
the last three lines of the
more.py
file listed as
Example 2-1
in the prior
chapter…

if __name__ == '__main__':                       # when run, not when imported
import sys
if len(sys.argv) == 1:                       # page stdin if no cmd args
more(sys.stdin.read())
else:
more(open(sys.argv[1]).read())

…it almost seems as if we should be able to redirect the standard
output of
teststreams.py
into
the standard input of
more.py
:

C:\...\PP4E\System\Streams>
python teststreams.py < input.txt | python ..\more.py
Hello stream world
Enter a number>8 squared is 64
Enter a number>6 squared is 36
Enter a number>Bye

This technique generally works for Python scripts. Here,
teststreams.py
takes input from a file again. And,
as in the last section, one Python program’s output is piped to
another’s input—the
more.py
script in the parent
(
..
) directory.

But there’s a subtle problem lurking in the preceding
more.py
command. Really, chaining worked there
only by sheer luck: if the first script’s output is long enough that
more
has to ask the user if it should
continue, the script will utterly fail (specifically, when
input
for user interaction triggers
EOFError
).

The problem is
that the augmented
more.py
uses
stdin
for two disjointed purposes. It
reads a reply from an interactive user on
stdin
by calling
input
, but now it
also
accepts the main input text on
stdin
.
When the
stdin
stream is really
redirected to an input file or pipe, we can’t use it to input a reply
from an interactive user; it contains only the text of the input source.
Moreover, because
stdin
is redirected
before the program even starts up, there is no way to know what it meant
prior to being redirected in the command line.

If we intend to accept input on
stdin
and
use the console
for user interaction, we have to do a bit more: we would also need to
use special interfaces to read user replies from a keyboard directly,
instead of standard input. On Windows, Python’s standard library
msvcrt
module provides such tools; on
many Unix-like platforms, reading from device file
/dev/tty
will usually suffice.

Since this is an arguably obscure use case, we’ll delegate a
complete solution to a suggested exercise.
Example 3-8
shows a Windows-only
modified version of the
more
script that pages the
standard input stream if called with no arguments, but also makes use of
lower-level and platform-specific tools to converse with a user at a
keyboard if needed.

"""
split and interactively page a string, file, or stream of
text to stdout; when run as a script, page stdin or file
whose name is passed on cmdline; if input is stdin, can't
use it for user reply--use platform-specific tools or GUI;
"""
import sys
def getreply():
"""
read a reply key from an interactive user
even if stdin redirected to a file or pipe
"""
if sys.stdin.isatty():                       # if stdin is console
return input('?')                        # read reply line from stdin
else:
if sys.platform[:3] == 'win':            # if stdin was redirected
import msvcrt                        # can't use to ask a user
msvcrt.putch(b'?')
key = msvcrt.getche()                # use windows console tools
msvcrt.putch(b'\n')                  # getch() does not echo key
return key
else:
assert False, 'platform not supported'
#linux?: open('/dev/tty').readline()[:-1]
def more(text, numlines=10):
"""
page multiline string to stdout
"""
lines = text.splitlines()
while lines:
chunk = lines[:numlines]
lines = lines[numlines:]
for line in chunk: print(line)
if lines and getreply() not in [b'y', b'Y']: break
if __name__ == '__main__':                       # when run, not when imported
if len(sys.argv) == 1:                       # if no command-line arguments
more(sys.stdin.read())                   # page stdin, no inputs
else:
more(open(sys.argv[1]).read())           # else page filename argument

Most of the new code in this version shows up in its
getreply
function. The file’s
isatty
method tells us whether
stdin
is connected to the console; if it is,
we simply read replies on
stdin
as
before. Of course, we have to add such extra logic only to scripts that
intend to interact with console users
and
take
input on
stdin
. In a GUI application,
for example, we could instead pop up dialogs, bind keyboard-press events
to run callbacks, and so on (we’ll meet GUIs in
Chapter 7
).

Armed with the reusable
getreply
function, though, we can safely run
our
moreplus
utility in a variety of
ways. As before, we can import and call this module’s function directly,
passing in whatever string we wish to page:

>>>
from moreplus import more
>>>
more(open('adderSmall.py').read())
import sys
print(sum(int(line) for line in sys.stdin))

Also as before, when run with a command-line
argument
, this script interactively pages through
the named file’s text:

C:\...\PP4E\System\Streams>
python moreplus.py adderSmall.py
import sys
print(sum(int(line) for line in sys.stdin))
C:\...\PP4E\System\Streams>
python moreplus.py moreplus.py
"""
split and interactively page a string, file, or stream of
text to stdout; when run as a script, page stdin or file
whose name is passed on cmdline; if input is stdin, can't
use it for user reply--use platform-specific tools or GUI;
"""
import sys
def getreply():
?
n

But now the script also correctly pages text redirected into
stdin
from either a
file
or a command
pipe
, even
if that text is too long to fit in a single display chunk. On most
shells, we send such input via redirection or pipe operators like
these:

C:\...\PP4E\System\Streams>
python moreplus.py < moreplus.py
"""
split and interactively page a string, file, or stream of
text to stdout; when run as a script, page stdin or file
whose name is passed on cmdline; if input is stdin, can't
use it for user reply--use platform-specific tools or GUI;
"""
import sys
def getreply():
?
n
C:\...\PP4E\System\Streams>
type moreplus.py | python moreplus.py
"""
split and interactively page a string, file, or stream of
text to stdout; when run as a script, page stdin or file
whose name is passed on cmdline; if input is stdin, can't
use it for user reply--use platform-specific tools or GUI;
"""
import sys
def getreply():
?
n

Finally, piping one Python script’s output into this script’s
input now works as expected, without botching user interaction (and not
just because we got lucky):

......\System\Streams>
python teststreams.py < input.txt | python moreplus.py
Hello stream world
Enter a number>8 squared is 64
Enter a number>6 squared is 36
Enter a number>Bye

Here, the standard
output
of one Python
script is fed to the standard
input
of another
Python script located in the same directory:
moreplus.py
reads the output of
teststreams.py
.

All of the redirections in such command lines work only because
scripts don’t care what standard input and output really are—interactive
users, files, or pipes between programs. For example, when run as a
script,
moreplus.py
simply reads stream
sys.stdin
; the command-line shell (e.g., DOS
on Windows, csh on Linux) attaches such streams to the source implied by
the command line before the script is started. Scripts use the preopened
stdin
and
stdout
file objects to access those sources,
regardless of their true nature.

And for readers keeping count, we have just run this single
more
pager script in four different
ways: by importing and calling its function, by passing a filename
command-line argument, by redirecting
stdin
to a file, and by piping a command’s
output to
stdin
. By supporting
importable functions, command-line arguments, and standard streams,
Python system tools code can be reused in a wide variety of
modes.

All of the previous
standard stream redirections work for programs written in
any language that hook into the standard streams and rely more on the
shell’s command-line processor than on Python itself. Command-line
redirection syntax like
<
filename
and
|
program
is evaluated by the shell,
not by Python. A more Pythonesque form of redirection can be done within
scripts themselves by resetting
sys.stdin
and
sys.stdout
to
file-like objects.

The main trick behind this mode is that anything that looks like a
file in terms of methods will work as a standard stream in Python. The
object’s interface (sometimes called its protocol), and not the object’s
specific datatype, is all that matters. That is:

Because
print
and
input
simply call the
write
and
readline
methods of whatever objects
sys.stdout
and
sys.stdin
happen to reference, we can use this
technique to both provide and intercept standard stream text with
objects implemented as classes.

If you’ve already studied Python, you probably know that such
plug-and-play compatibility is usually called
polymorphism
—
it doesn’t matter what an object is, and it doesn’t matter
what its interface does, as long as it provides the expected interface.
This liberal approach to datatypes accounts for much of the conciseness
and flexibility of Python code. Here, it provides a way for scripts to
reset their own streams.
Example 3-9
shows a utility module that
demonstrates this concept.

"""
file-like objects that save standard output text in a string and provide
standard input text from a string; redirect runs a passed-in function
with its output and input streams reset to these file-like class objects;
"""
import sys                                      # get built-in modules
class Output:                                   # simulated output file
def __init__(self):
self.text = ''                          # empty string when created
def write(self, string):                    # add a string of bytes
self.text += string
def writelines(self, lines):                # add each line in a list
for line in lines: self.write(line)
class Input:                                    # simulated input file
def __init__(self, input=''):               # default argument
self.text = input                       # save string when created
def read(self, size=None):                  # optional argument
if size == None:                        # read N bytes, or all
res, self.text = self.text, ''
else:
res, self.text = self.text[:size], self.text[size:]
return res
def readline(self):
eoln = self.text.find('\n')             # find offset of next eoln
if eoln == −1:                          # slice off through eoln
res, self.text = self.text, ''
else:
res, self.text = self.text[:eoln+1], self.text[eoln+1:]
return res
def redirect(function, pargs, kargs, input):    # redirect stdin/out
savestreams = sys.stdin, sys.stdout         # run a function object
sys.stdin   = Input(input)                  # return stdout text
sys.stdout  = Output()
try:
result = function(*pargs, **kargs)      # run function with args
output = sys.stdout.text
finally:
sys.stdin, sys.stdout = savestreams     # restore if exc or not
return (result, output)                     # return result if no exc

This module defines two classes that masquerade as real
files:

The
redirect
function at the
bottom of this file combines these two objects to run a single function
with input and output redirected entirely to Python class objects. The
passed-in function to run need not know or care that its
print
and
input
function calls and
stdin
and
stdout
method calls are talking to a class
rather than to a real file, pipe, or user.

To demonstrate, import and run the
interact
function at the heart of the
teststreams
script of
Example 3-5
that we’ve been running
from the shell (to use the
redirection
utility function, we need to
deal in terms of functions, not files). When run directly, the function
reads from the keyboard and writes to the screen, just as if it were run
as a program without redirection:

C:\...\PP4E\System\Streams>
python
>>>
from teststreams import interact
>>>
interact()
Hello stream world
Enter a number>
2
2 squared is 4
Enter a number>
3
3 squared is 9
Enter a number^Z
Bye
>>>

Now, let’s run this function under the control of the redirection
function in
redirect.py
and
pass in some canned input text. In this mode, the
interact
function takes its input from the
string we pass in (
'4\n5\n6\n'
—three
lines with explicit end-of-line characters), and the result of running
the function is a tuple with its return value plus a string containing
all the text written to the standard output stream:

>>>
from redirect import redirect
>>>
(result, output) = redirect(interact, (), {}, '4\n5\n6\n')
>>>
print(result)
None
>>>
output
'Hello stream world\nEnter a number>4 squared is 16\nEnter a number>5 squared
is 25\nEnter a number>6 squared is 36\nEnter a number>Bye\n'

The output is a single, long string containing the concatenation
of all text written to standard output. To make this look better, we can
pass it to
print
or split it up with
the string object’s
splitlines
method:

>>>
for line in output.splitlines(): print(line)
...
Hello stream world
Enter a number>4 squared is 16
Enter a number>5 squared is 25
Enter a number>6 squared is 36
Enter a number>Bye

Better still, we can reuse the
more.py
module we wrote in the preceding
chapter (
Example 2-1
); it’s less to type
and remember, and it’s already known to work well (the following, like
all cross-directory imports in this book’s examples, assumes that the
directory containing the
PP4E
root is
on your module search path—change your
PYTHONPATH
setting as needed):

>>>
from PP4E.System.more import more
>>>
more(output)
Hello stream world
Enter a number>4 squared is 16
Enter a number>5 squared is 25
Enter a number>6 squared is 36
Enter a number>Bye

This is an artificial example, of course, but the techniques
illustrated are widely applicable. For instance, it’s straightforward to
add a GUI interface to a program written to interact with a command-line
user. Simply intercept standard output with an object such as the
Output
class instance shown earlier
and throw the text string up in a window. Similarly, standard input can
be reset to an object that fetches text from a graphical interface
(e.g., a popped-up dialog box). Because classes are plug-and-play
compatible with real files, we can use them in any tool that expects a
file. Watch for a GUI stream-redirection module named
guiStreams
in
Chapter 10
that provides a concrete
implementation of some of these
ideas.