Programming Python (38 page)

One fine point before
we move on: notice the seemingly superfluous exception
handling in
Example 6-4
’s
tryprint
function. When I first tried
to scan an entire drive as shown in the preceding section, this script
died on a Unicode encoding error while trying to print a directory name
of a saved web page. Adding the exception handler skips the error
entirely.

This demonstrates a subtle but pragmatically important issue:
Python 3.X’s Unicode orientation extends to filenames, even if they are
just printed. As we learned in
Chapter 4
, because filenames may contain
arbitrary text,
os.listdir
returns filenames in two different ways—we get back
decoded Unicode strings when we pass in a normal
str
argument, and still-encoded byte strings
when we send a
bytes
:

>>>
import os
>>>
os.listdir('.')[:4]
['bigext-tree.py', 'bigpy-dir.py', 'bigpy-path.py', 'bigpy-tree.py']
>>>
os.listdir(b'.')[:4]
[b'bigext-tree.py', b'bigpy-dir.py', b'bigpy-path.py', b'bigpy-tree.py']

Both
os.walk
(used in the
Example 6-4
script) and
glob.glob
inherit this behavior for the
directory and file names they return, because they work by calling
os.listdir
internally at each
directory level. For all these calls, passing in a byte string argument
suppresses Unicode decoding of file and directory names. Passing a
normal string assumes that filenames are decodable per the file system’s
Unicode scheme.

The reason this potentially mattered to this section’s example is
that running the tree search version over an entire hard drive
eventually reached an undecodable filename (an old saved web page with
an odd name), which generated an exception when the
print
function tried to display it. Here’s a
simplified recreation of the error, run in a shell window (Command
Prompt) on Windows:

>>>
root = r'C:\py3000'
>>>
for (dir, subs, files) in os.walk(root): print(dir)
...
C:\py3000
C:\py3000\FutureProofPython - PythonInfo Wiki_files
C:\py3000\Oakwinter_com  Code » Porting setuptools to py3k_files
Traceback (most recent call last):
File "", line 1, in 
File "C:\Python31\lib\encodings\cp437.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2019' in position
45: character maps to 

One way out of this dilemma is to use
bytes
strings for the directory root name—this
suppresses filename decoding in the
os.listdir
calls run by
os.walk
, and effectively limits the scope of
later printing to raw bytes. Since printing does not have to deal with
encodings, it works without error. Manually encoding to bytes prior to
printing works too, but the results are slightly different:

>>>
root.encode()
b'C:\\py3000'
>>>
for (dir, subs, files) in os.walk(root.encode()): print(dir)
...
b'C:\\py3000'
b'C:\\py3000\\FutureProofPython - PythonInfo Wiki_files'
b'C:\\py3000\\Oakwinter_com  Code \xbb Porting setuptools to py3k_files'
b'C:\\py3000\\What\x92s New in Python 3_0 \x97 Python Documentation'
>>>
for (dir, subs, files) in os.walk(root): print(dir.encode())
...
b'C:\\py3000'
b'C:\\py3000\\FutureProofPython - PythonInfo Wiki_files'
b'C:\\py3000\\Oakwinter_com  Code \xc2\xbb Porting setuptools to py3k_files'
b'C:\\py3000\\What\xe2\x80\x99s New in Python 3_0 \xe2\x80\x94 Python Documentation'

Unfortunately, either approach means that all the directory names
printed during the walk display as cryptic byte strings. To maintain the
better readability of normal strings, I instead opted for the exception
handler approach used in the script’s code. This avoids the issues
entirely:

>>>
for (dir, subs, files) in os.walk(root):
...
try:
...
print(dir)
...
except UnicodeEncodeError:
...
print(dir.encode())
# or simply punt if enocde may fail too
...
C:\py3000
C:\py3000\FutureProofPython - PythonInfo Wiki_files
C:\py3000\Oakwinter_com  Code » Porting setuptools to py3k_files
b'C:\\py3000\\What\xe2\x80\x99s New in Python 3_0 \xe2\x80\x94 Python Documentation'

Oddly, though, the error seems more related to printing than to
Unicode encodings of filenames—because the filename did not fail until
printed, it must have been decodable when its string was created
initially. That’s why wrapping up the
print
in a
try
suffices; otherwise, the error would occur
earlier.

Moreover, this error does not occur if the script’s output is
redirected to a file, either at the shell level (
bigext-tree.py c:\ > out
), or by the print
call itself (
print(dir, file=F)
). In
the latter case the output file must later be read back in binary mode,
as text mode triggers the same error when printing the file’s content to
the shell window (but again, not until printed). In fact, the exact same
code that fails when run in a system shell Command Prompt on Windows
works without error when run in the IDLE GUI on the same platform—the
tkinter GUI used by IDLE handles display of characters that printing to
standard output connected to a shell terminal window does not:

>>>
import os
# run in IDLE (a tkinter GUI), not system shell
>>>
root = r'C:\py3000'
>>>
for (dir, subs, files) in os.walk(root): print(dir)
C:\py3000
C:\py3000\FutureProofPython - PythonInfo Wiki_files
C:\py3000\Oakwinter_com  Code » Porting setuptools to py3k_files
C:\py3000\What's New in Python 3_0 — Python Documentation_files

In other words, the exception occurs only when printing to a shell
window, and long after the file name string is created. This reflects an
artifact of extra translations
performed
by the Python printer, not of
Unicode file names in general. Because we have no room for further
exploration here, though, we’ll have to be satisfied with the fact that
our exception handler sidesteps the printing problem altogether. You
should still be aware of the implications of Unicode filename decoding,
though; on some platforms you may need to pass byte strings to
os.walk
in this script to prevent decoding
errors as filenames are
created.
^[
18
]

Since Unicode is still relatively new in 3.1, be sure to test for
such errors on your computer and your Python. Also see also Python’s
manuals for more on the treatment of Unicode filenames, and the text
Learning Python
for more on Unicode in general. As
noted earlier, our scripts also had to open text files in binary mode
because some might contain undecodable
content
too.
It might seem surprising that Unicode issues can crop up in basic
printing like this too, but such is life in the brave new Unicode world.
Many real-world scripts don’t need to care much about Unicode, of
course—including those we’ll explore in the next
section.

Since all the computers
in my house ran Python, a simple portable Python script
came to the rescue. The Python program in
Example 6-5
distributes a single
file’s contents among a set of part files and stores those part files in
a directory.

#!/usr/bin/python
"""
################################################################################
split a file into a set of parts; join.py puts them back together;
this is a customizable version of the standard Unix split command-line
utility; because it is written in Python, it also works on Windows and
can be easily modified; because it exports a function, its logic can
also be imported and reused in other applications;
################################################################################
"""
import sys, os
kilobytes = 1024
megabytes = kilobytes * 1000
chunksize = int(1.4 * megabytes)                   # default: roughly a floppy
def split(fromfile, todir, chunksize=chunksize):
if not os.path.exists(todir):                  # caller handles errors
os.mkdir(todir)                            # make dir, read/write parts
else:
for fname in os.listdir(todir):            # delete any existing files
os.remove(os.path.join(todir, fname))
partnum = 0
input = open(fromfile, 'rb')                   # binary: no decode, endline
while True:                                    # eof=empty string from read
chunk = input.read(chunksize)              # get next part <= chunksize
if not chunk: break
partnum += 1
filename = os.path.join(todir, ('part%04d' % partnum))
fileobj  = open(filename, 'wb')
fileobj.write(chunk)
fileobj.close()                            # or simply open().write()
input.close()
assert partnum <= 9999                         # join sort fails if 5 digits
return partnum
if __name__ == '__main__':
if len(sys.argv) == 2 and sys.argv[1] == '-help':
print('Use: split.py [file-to-split target-dir [chunksize]]')
else:
if len(sys.argv) < 3:
interactive = True
fromfile = input('File to be split? ')           # input if clicked
todir    = input('Directory to store part files? ')
else:
interactive = False
fromfile, todir = sys.argv[1:3]                  # args in cmdline
if len(sys.argv) == 4: chunksize = int(sys.argv[3])
absfrom, absto = map(os.path.abspath, [fromfile, todir])
print('Splitting', absfrom, 'to', absto, 'by', chunksize)
try:
parts = split(fromfile, todir, chunksize)
except:
print('Error during split:')
print(sys.exc_info()[0], sys.exc_info()[1])
else:
print('Split finished:', parts, 'parts are in', absto)
if interactive: input('Press Enter key') # pause if clicked

By default, this script splits the input file into chunks that are
roughly the size of a floppy disk—perfect for moving big files between
the electronically isolated machines of the time. Most importantly,
because this is all portable Python code, this script will run on just
about any machine, even ones without their own file splitter. All it
requires is an installed Python. Here it is at work splitting a Python
3.1 self-installer executable located in the current working directory
on Windows (I’ve omitted a few
dir
output lines to save space here; use
ls
-l
on Unix):

C:\temp>
cd C:\temp
C:\temp>
dir python-3.1.msi
...more...
06/27/2009  04:53 PM        13,814,272 python-3.1.msi
1 File(s)     13,814,272 bytes
0 Dir(s)  188,826,189,824 bytes free
C:\temp>
python C:\...\PP4E\System\Filetools\split.py -help
Use: split.py [file-to-split target-dir [chunksize]]
C:\temp>
python C:\...\P4E\System\Filetools\split.py python-3.1.msi pysplit
Splitting C:\temp\python-3.1.msi to C:\temp\pysplit by 1433600
Split finished: 10 parts are in C:\temp\pysplit
C:\temp>
dir pysplit
...more...
02/21/2010  11:13 AM              .
02/21/2010  11:13 AM              ..
02/21/2010  11:13 AM         1,433,600 part0001
02/21/2010  11:13 AM         1,433,600 part0002
02/21/2010  11:13 AM         1,433,600 part0003
02/21/2010  11:13 AM         1,433,600 part0004
02/21/2010  11:13 AM         1,433,600 part0005
02/21/2010  11:13 AM         1,433,600 part0006
02/21/2010  11:13 AM         1,433,600 part0007
02/21/2010  11:13 AM         1,433,600 part0008
02/21/2010  11:13 AM         1,433,600 part0009
02/21/2010  11:13 AM           911,872 part0010
10 File(s)     13,814,272 bytes
2 Dir(s)  188,812,328,960 bytes free

Each of these generated part files represents one binary chunk of
the file
python-3.1.msi
—
a
chunk small enough to fit comfortably on a floppy disk of the time. In
fact, if you add the sizes of the generated part files given by the
ls
command, you’ll come up with
exactly the same number of bytes as the original file’s size. Before we
see how to put these files back together again, here are a few points to
ponder as you study this script’s code:

Back to moving big files around
the house: after downloading a big game program file, you
can run the previous splitter script by clicking on its name in Windows
Explorer and typing filenames. After a split, simply copy each part file
onto its own floppy (or other more modern medium), walk the files to the
destination machine, and re-create the split output directory on the
target computer by copying the part files. Finally, the script in
Example 6-6
is clicked or otherwise
run to put the parts back together.

#!/usr/bin/python
"""
################################################################################
join all part files in a dir created by split.py, to re-create file.
This is roughly like a 'cat fromdir/* > tofile' command on unix, but is
more portable and configurable, and exports the join operation as a
reusable function.  Relies on sort order of filenames: must be same
length.  Could extend split/join to pop up Tkinter file selectors.
################################################################################
"""
import os, sys
readsize = 1024
def join(fromdir, tofile):
output = open(tofile, 'wb')
parts  = os.listdir(fromdir)
parts.sort()
for filename in parts:
filepath = os.path.join(fromdir, filename)
fileobj  = open(filepath, 'rb')
while True:
filebytes = fileobj.read(readsize)
if not filebytes: break
output.write(filebytes)
fileobj.close()
output.close()
if __name__ == '__main__':
if len(sys.argv) == 2 and sys.argv[1] == '-help':
print('Use: join.py [from-dir-name to-file-name]')
else:
if len(sys.argv) != 3:
interactive = True
fromdir = input('Directory containing part files? ')
tofile  = input('Name of file to be recreated? ')
else:
interactive = False
fromdir, tofile = sys.argv[1:]
absfrom, absto = map(os.path.abspath, [fromdir, tofile])
print('Joining', absfrom, 'to make', absto)
try:
join(fromdir, tofile)
except:
print('Error joining files:')
print(sys.exc_info()[0], sys.exc_info()[1])
else:
print('Join complete: see', absto)
if interactive: input('Press Enter key') # pause if clicked

Here is a join in progress on Windows, combining the split files
we made a moment ago; after running the
join
script, you still may need to run
something like
zip
,
gzip
, or
tar
to unpack an archive file unless it’s
shipped as an executable, but at least the original downloaded file is
set to go
^[
20
]
:

C:\temp>
python C:\...\PP4E\System\Filetools\join.py -help
Use: join.py [from-dir-name to-file-name]
C:\temp>
python C:\...\PP4E\System\Filetools\join.py pysplit mypy31.msi
Joining C:\temp\pysplit to make C:\temp\mypy31.msi
Join complete: see C:\temp\mypy31.msi
C:\temp>
dir *.msi
...more...
02/21/2010  11:21 AM        13,814,272 mypy31.msi
06/27/2009  04:53 PM        13,814,272 python-3.1.msi
2 File(s)     27,628,544 bytes
0 Dir(s)  188,798,611,456 bytes free
C:\temp>
fc /b mypy31.msi python-3.1.msi
Comparing files mypy31.msi and PYTHON-3.1.MSI
FC: no differences encountered

The join script simply uses
os.listdir
to collect
all the part files in a directory created by
split
, and sorts the filename list to put the
parts back together in the correct order. We get back an exact
byte-for-byte copy of the original file (proved by the DOS
fc
command in the code; use
cmp
on Unix).

Some of this process is still manual, of course (I never did
figure out how to script the “walk the floppies to your bedroom” step),
but the
split
and
join
scripts make it both quick and simple to
move big files around. Because this script is also portable Python code,
it runs on any platform to which we cared to move split files. For
instance, my home computers ran both Windows and Linux at the time;
since this script runs on either platform, the gamers were covered.
Before we move on, here are a couple of implementation details worth
underscoring in the
join
script’s
code: