Programming Python (171 page)

This part of the book presents a collection of additional Python
application topics. Most of the tools presented along the way can be used
in a wide variety of application domains. You’ll find the following
chapters here:

This is the last technical part of the book, and it makes heavy use
of tools presented earlier in the text to help underscore the notion of
code reuse. For instance, a calculator GUI (PyCalc) serves to demonstrate
language processing and code reuse concepts.

So far in this book, we’ve used Python in the system programming,
GUI development, and Internet scripting domains—three of Python’s most
common applications, and representative of its use as an application
programming language at large. In the next four chapters, we’re going to
take a quick look at other major Python programming topics: persistent
data, data structure techniques, text and language processing, and
Python/C integration.

These four topics are not really application areas themselves, but
they are techniques that span domains. The database topics in this
chapter, for instance, can be applied on the Web, in desktop GUI
applications, and so on. Text processing is a similarly general tool.
Moreover, none of these final four topics is covered exhaustively (each
could easily fill a book alone), but we’ll sample Python in action in
these domains and highlight their core concepts and tools. If any of these
chapters spark your interest, additional resources are readily available
in the Python world.

Flat files

Text and bytes stored directly on your computer

DBM keyed files

Keyed access to
strings stored in dictionary-like files

Pickled objects

Serialized Python
objects saved to files and streams

Shelve files

Pickled Python
objects saved in DBM keyed files

Object-oriented databases (OODBs)

Persistent
Python objects stored in persistent dictionaries
(ZODB, Durus)

SQL relational databases (RDBMSs)

Table-based
storage that supports SQL queries (SQLite, MySQL,
PostGreSQL, etc.)

Object relational mappers (ORMs)

Mediators
that map Python classes to relational
tables (SQLObject, SQLAlchemy)

Fourth edition coverage note
: The prior
edition of this book used the
mysql-python
interface to the MySQL relational
database system, as well as the ZODB object database system. As I update
this chapter in June 2010, neither of these is yet available for Python
3.X, the version of Python used in this edition. Because of that, most
ZODB information has been trimmed, and the SQL database examples here
were changed to use the SQLite in-process database system that ships
with Python 3.X as part of its standard library. The prior edition’s
ZODB and MySQL examples and overviews are still available in the
examples package, as described later. Because Python’s SQL database API
is portable, though, the SQLite code here should work largely unchanged
on most other systems.

Although
DBM filesystems have to do a bit of work to map chunks of
stored data to keys for fast retrieval (technically, they generally use
a technique
called
hashing
to store data in
files), your scripts don’t need to care about the action going on behind
the scenes. In fact, DBM is one of the easiest ways to save information
in Python—DBM files behave so much like in-memory dictionaries that you
may forget you’re actually dealing with a file at all. For instance,
given a DBM file object:

DBM file objects also support common dictionary methods such as
keys-list fetches and tests and key deletions. The DBM library itself is
hidden behind this simple model. Since it is so simple, let’s jump right
into an interactive example that creates a DBM file and shows how the
interface works:

C:\...\PP4E\Dbase>
python
>>>
import dbm
# get interface: bsddb, gnu, ndbm, dumb
>>>
file = dbm.open('movie', 'c')
# make a DBM file called 'movie'
>>>
file['Batman'] = 'Pow!'
# store a string under key 'Batman'
>>>
file.keys()
# get the file's key directory
[b'Batman']
>>>
file['Batman']
# fetch value for key 'Batman'
b'Pow!'
>>>
who  = ['Robin', 'Cat-woman', 'Joker']
>>>
what = ['Bang!', 'Splat!', 'Wham!']
>>>
for i in range(len(who)):
...
file[who[i]] = what[i]
# add 3 more "records"
...
>>>
file.keys()
[b'Cat-woman', b'Batman', b'Joker', b'Robin']
>>>
len(file), 'Robin' in file, file['Joker']
(4, True, b'Wham!')
>>>
file.close()
# close sometimes required

Internally, importing the
dbm
standard library
module automatically loads whatever DBM interface is available in your
Python interpreter (attempting alternatives in a fixed order), and
opening the new DBM file creates one or more external files with names
that start with the string
'movie'
(more on the details in a moment). But after the import and open, a DBM
file is virtually indistinguishable from a dictionary.

In effect, the object called
file
here can be thought of as a dictionary
mapped to an external file called
movie
; the only obvious differences are that
keys must be strings (not arbitrary immutables), and we need to remember
to open to access and close after changes.

Unlike normal dictionaries, though, the contents of
file
are retained between Python program runs.
If we come back later and restart Python, our dictionary is still
available. Again, DBM files are like dictionaries that must be
opened:

C:\...\PP4E\Dbase>
python
>>>
import dbm
>>>
file = dbm.open('movie', 'c')
# open existing DBM file
>>>
file['Batman']
b'Pow!'
>>>
file.keys()
# keys gives an index list
[b'Cat-woman', b'Batman', b'Joker', b'Robin']
>>>
for key in file.keys(): print(key, file[key])
...
b'Cat-woman' b'Splat!'
b'Batman' b'Pow!'
b'Joker' b'Wham!'
b'Robin' b'Bang!'

Notice how DBM files return a real list for the
keys
call; not shown here, their
values
method instead returns an iterable view
like dictionaries. Further, DBM files always store both keys and values
as
bytes
objects; interpretation as
arbitrary types of Unicode text is left to the client application. We
can use either
bytes
or
str
strings in our code when accessing or
storing keys and values—using
bytes
allows your keys and values to retain arbitrary Unicode encodings, but
str
objects in our code will be
encoded to
bytes
internally using the
UTF-8 Unicode encoding by Python’s DBM implementation.

Still, we can always decode to Unicode
str
strings to display in a more friendly
fashion if desired, and DBM files have a keys iterator just like
dictionaries. Moreover, assigning and deleting keys updates the DBM
file, and we should close after making changes (this ensure that changes
are flushed to disk):

>>>
for key in file: print(key.decode(), file[key].decode())
...
Cat-woman Splat!
Batman Pow!
Joker Wham!
Robin Bang!
>>>
file['Batman'] = 'Ka-Boom!'
# change Batman slot
>>>
del file['Robin']
# delete the Robin entry
>>>
file.close()
# close it after changes

Apart from having to import the interface and open and close the
DBM file, Python programs don’t have to know anything about DBM itself.
DBM modules achieve this integration by overloading the indexing
operations and routing them to more primitive library tools. But you’d
never know that from looking at this Python code—DBM files look like
normal Python dictionaries, stored on external files. Changes made to
them are retained indefinitely:

C:\...\PP4E\Dbase>
python
>>>
import dbm
# open DBM file again
>>>
file = dbm.open('movie', 'c')
>>>
for key in file: print(key.decode(), file[key].decode())
...
Cat-woman Splat!
Batman Ka-Boom!
Joker Wham!

As you can see, this is about as simple as it can be.
Table 17-1
lists the most commonly used DBM file
operations. Once such a file is opened, it is processed just as though
it were an in-memory Python dictionary. Items are fetched by indexing
the file object by key and are stored by assigning to a
key.

Despite the dictionary-like interface, DBM files really do map to
one or more external files. For instance, the underlying default
dbm
interface used by Python 3.1 on
Windows writes two files—
movie.dir
and
movie.dat
—when a DBM file called
movie
is made, and saves a
movie.bak
on later opens. If your Python has
access to a different underlying keyed-file interface, different
external files might show up on your computer.

Technically, the module
dbm
is
really an interface to whatever DBM-like filesystem you
have available in your Python:

Future Pythons are free to change this selection order, and may
even add additional alternatives to it. You normally don’t need to care
about any of this, though, unless you delete any of the files your DBM
creates, or transfer them between machines with different
configurations—if you need to care about the
portability
of your DBM files (and as we’ll see
later, by proxy, that of your shelve files), you should configure
machines such that all have the same DBM interface installed or rely
upon the
dumb
fallback. For example,
the Berkeley DB package (a.k.a.
bsddb
) used by
dbm.bsd
is widely available and
portable.

Note that DBM files may or may not need to be explicitly closed,
per the last entry in
Table 17-1
. Some DBM
files don’t require a close call, but some depend on it to flush changes
out to disk. On such systems, your file may be corrupted if you omit the
close call. Unfortunately, the default DBM in some older Windows
Pythons,
dbhash
(a.k.a.
bsddb
), is one of the DBM systems that
requires a close call to avoid data loss. As a rule of thumb, always
close your DBM files explicitly after making changes and before your
program exits to avoid potential problems; it’s essential a “commit”
operation for these files. This rule extends by proxy to shelves, a
topic we’ll meet later in this
chapter.