Programming Python (6 page)

As mentioned
earlier, one potential disadvantage of this section’s
examples so far is that they may become slow for very large databases:
because the entire database must be loaded and rewritten to update a
single record, this approach can waste time. We could improve on this by
storing each record in the database in a separate flat file. The next
three examples show one way to do so;
Example 1-8
stores each record in its
own flat file, using each record’s original key as its filename with a
.pkl
appended (it creates the files
bob.pkl, sue.pkl
, and
tom.pkl
in the current working directory).

from initdata import bob, sue, tom
import pickle
for (key, record) in [('bob', bob), ('tom', tom), ('sue', sue)]:
recfile = open(key + '.pkl', 'wb')
pickle.dump(record, recfile)
recfile.close()

Next,
Example 1-9
dumps
the entire database by using the standard library’s
glob
module to do filename expansion and thus collect all the
files in this directory with a
.pkl
extension. To
load a single record, we open its file and deserialize with
pickle
; we must load only one record file,
though, not the entire database, to fetch one record.

import pickle, glob
for filename in glob.glob('*.pkl'):         # for 'bob','sue','tom'
recfile = open(filename, 'rb')
record  = pickle.load(recfile)
print(filename, '=>\n  ', record)
suefile = open('sue.pkl', 'rb')
print(pickle.load(suefile)['name'])         # fetch sue's name

Finally,
Example 1-10
updates the database by fetching a record from its file, changing it in
memory, and then writing it back to its pickle file. This time, we have
to fetch and rewrite only a single record file, not the full database,
to update.

import pickle
suefile = open('sue.pkl', 'rb')
sue = pickle.load(suefile)
suefile.close()
sue['pay'] *= 1.10
suefile = open('sue.pkl', 'wb')
pickle.dump(sue, suefile)
suefile.close()

Here are our file-per-record scripts in action; the results are
about the same as in the prior section, but database keys become real
filenames now. In a sense, the filesystem becomes our top-level
dictionary—filenames provide direct access to each
record.

...\PP4E\Preview>
python make_db_pickle_recs.py
...\PP4E\Preview>
python dump_db_pickle_recs.py
bob.pkl =>
{'pay': 30000, 'job': 'dev', 'age': 42, 'name': 'Bob Smith'}
sue.pkl =>
{'pay': 40000, 'job': 'hdw', 'age': 45, 'name': 'Sue Jones'}
tom.pkl =>
{'pay': 0, 'job': None, 'age': 50, 'name': 'Tom'}
Sue Jones
...\PP4E\Preview>
python update_db_pickle_recs.py
...\PP4E\Preview>
python dump_db_pickle_recs.py
bob.pkl =>
{'pay': 30000, 'job': 'dev', 'age': 42, 'name': 'Bob Smith'}
sue.pkl =>
{'pay':
44000.0
, 'job': 'hdw', 'age': 45, 'name': 'Sue Jones'}
tom.pkl =>
{'pay': 0, 'job': None, 'age': 50, 'name': 'Tom'}
Sue Jones

Pickling
objects to files, as shown in the preceding section, is an
optimal scheme in many applications. In fact, some applications use
pickling of Python objects across network sockets as a simpler
alternative to network protocols such as the SOAP and XML-RPC web
services architectures (also supported by Python, but much heavier than
pickle
).

Moreover, assuming your filesystem can handle as many files as
you’ll need, pickling one record per file also obviates the need to load
and store the entire database for each update. If we really want keyed
access to records, though, the Python standard library offers an even
higher-level tool: shelves.

Shelves automatically pickle objects to and from a keyed-access
filesystem. They behave much like dictionaries that must be opened, and
they persist after each program exits. Because they give us key-based
access to stored records, there is no need to manually manage one flat
file per record—the shelve system automatically splits up stored records
and fetches and updates only those records that are accessed and
changed. In this way, shelves provide utility similar to per-record
pickle files, but they are usually easier to code.

The
shelve
interface is just as
simple as
pickle
: it is identical to
dictionaries, with extra open and close calls. In fact, to your code, a
shelve really does appear to be a persistent dictionary of persistent
objects; Python does all the work of mapping its content to and from a
file. For instance,
Example 1-11
shows how to store our in-memory dictionary objects in a shelve for
permanent keeping.

from initdata import bob, sue
import shelve
db = shelve.open('people-shelve')
db['bob'] = bob
db['sue'] = sue
db.close()

This script creates one or more files in the current directory
with the name
people-shelve
as a prefix (in Python
3.1 on Windows,
people-shelve.bak
,
people-shelve.dat
, and
people-shelve.dir
). You shouldn’t delete
these files (they are your database!), and you should be sure to use the
same base name in other scripts that access the shelve.
Example 1-12
, for instance,
reopens the shelve and indexes it by key to fetch its stored
records.

import shelve
db = shelve.open('people-shelve')
for key in db:
print(key, '=>\n  ', db[key])
print(db['sue']['name'])
db.close()

We still have a dictionary of dictionaries here, but the top-level
dictionary is really a shelve mapped onto a file. Much happens when you
access a shelve’s keys—it uses
pickle
internally to serialize and deserialize objects stored, and it
interfaces with a keyed-access filesystem. From your perspective,
though, it’s just a persistent dictionary.
Example 1-13
shows how to code shelve
updates.

from initdata import tom
import shelve
db = shelve.open('people-shelve')
sue = db['sue']                       # fetch sue
sue['pay'] *= 1.50
db['sue'] = sue                       # update sue
db['tom'] = tom                       # add a new record
db.close()

Notice how this code fetches
sue
by key, updates in memory, and then
reassigns to the key to update the shelve; this is a requirement of
shelves by default, but not always of more advanced shelve-like systems
such as ZODB, covered in
Chapter 17
.
As we’ll see later
,
shelve.open
also has
a newer
writeback
keyword argument,
which, if passed
True
, causes all
records loaded from the shelve to be cached in memory, and automatically
written back to the shelve when it is closed; this avoids manual write
backs on changes, but can consume memory and make closing slow.

Also note how shelve files are explicitly closed. Although we
don’t need to pass mode flags to
shelve.open
(by default it creates the shelve
if needed, and opens it for reads and writes otherwise), some underlying
keyed-access filesystems may require a
close
call in order to flush output buffers
after changes.

Finally, here are the shelve-based scripts on the job, creating,
changing, and fetching records. The records are still dictionaries, but
the database is now a dictionary-like shelve which automatically retains
its state in a file between program runs:

...\PP4E\Preview>
python make_db_shelve.py
...\PP4E\Preview>
python dump_db_shelve.py
bob =>
{'pay': 30000, 'job': 'dev', 'age': 42, 'name': 'Bob Smith'}
sue =>
{'pay': 40000, 'job': 'hdw', 'age': 45, 'name': 'Sue Jones'}
Sue Jones
...\PP4E\Preview>
python update_db_shelve.py
...\PP4E\Preview>
python dump_db_shelve.py
bob =>
{'pay': 30000, 'job': 'dev', 'age': 42, 'name': 'Bob Smith'}
sue =>
{'pay':
60000.0
, 'job': 'hdw', 'age': 45, 'name': 'Sue Jones'}
tom =>
{'pay': 0, 'job': None, 'age': 50, 'name': 'Tom'}
Sue Jones

When we ran the update and dump scripts here, we added a new
record for key
tom
and increased
Sue’s pay field by 50 percent. These changes are permanent because the
record dictionaries are mapped to an external file by shelve. (In fact,
this is a particularly good script for Sue—something she might consider
scheduling to run often, using a cron job on Unix, or a Startup folder
or msconfig entry on
Windows…)

Structure

With OOP, we can naturally associate processing logic with
record data—classes provide both a program unit that combines logic
and data in a single package and a hierarchy that allows code to be
easily factored to avoid redundancy.

Encapsulation

With OOP, we can also wrap up details such as name processing
and pay increases behind method functions—i.e., we are free to
change method implementations without breaking their users.

Customization

And with OOP, we have a natural growth path. Classes can be
extended and customized by coding new subclasses, without changing
or breaking already working code.

OOP is easy to
use in Python, thanks largely to Python’s dynamic typing
model. In fact, it’s so easy that we’ll jump right into an example:
Example 1-14
implements our
database records as class instances rather than as dictionaries.

class Person:
def __init__(self, name, age, pay=0, job=None):
self.name = name
self.age  = age
self.pay  = pay
self.job  = job
if __name__ == '__main__':
bob = Person('Bob Smith', 42, 30000, 'software')
sue = Person('Sue Jones', 45, 40000, 'hardware')
print(bob.name, sue.pay)
print(bob.name.split()[-1])
sue.pay *= 1.10
print(sue.pay)

There is not much to this class—just a constructor method that
fills out the instance with data passed in as arguments to the class
name. It’s sufficient to represent a database record, though, and it can
already provide tools such as defaults for pay and job fields that
dictionaries cannot. The self-test code at the bottom of this file
creates two instances (records) and accesses their attributes (fields);
here is this file’s output when run under IDLE (a system command-line
works just as well):

Bob Smith 40000
Smith
44000.0

This isn’t a database yet, but we could stuff these objects into a
list or dictionary as before in order to collect them as a unit:

>>>
from person_start import Person
>>>
bob = Person('Bob Smith', 42)
>>>
sue = Person('Sue Jones', 45, 40000)
>>>
people = [bob, sue]
# a "database" list
>>>
for person in people:
print(person.name, person.pay)
Bob Smith 0
Sue Jones 40000
>>>
x = [(person.name, person.pay) for person in people]
>>>
x
[('Bob Smith', 0), ('Sue Jones', 40000)]
>>>
[rec.name for rec in people if rec.age >= 45]
# SQL-ish query
['Sue Jones']
>>>
[(rec.age ** 2 if rec.age >= 45 else rec.age) for rec in people]
[42, 2025]

Notice that Bob’s pay defaulted to zero this time because we
didn’t pass in a value for that argument (maybe Sue is supporting him
now?). We might also implement a class that represents the database,
perhaps as a subclass of the built-in list or dictionary types, with
insert and delete methods that encapsulate the way the database is
implemented. We’ll abandon this path for now, though, because it will be
more useful to store these records persistently in a shelve, which
already encapsulates stores and fetches behind an interface for us.
Before we do, though, let’s add some
logic.

So far, our
class is just data: it replaces dictionary keys with
object attributes, but it doesn’t add much to what we had before. To
really leverage the power of classes, we need to add some behavior. By
wrapping up bits of behavior in class method functions, we can insulate
clients from changes. And by packaging methods in classes along with
data, we provide a natural place for readers to look for code. In a
sense, classes combine records and the programs that process those
records; methods provide logic that interprets and updates the data (we
say they are
object-oriented
, because they always
process an object’s data).

For instance,
Example 1-15
adds the last-name and raise logic as class methods; methods use the
self
argument to access or update the
instance (record) being processed.

class Person:
def __init__(self, name, age, pay=0, job=None):
self.name = name
self.age  = age
self.pay  = pay
self.job  = job
def lastName(self):
return self.name.split()[-1]
def giveRaise(self, percent):
self.pay *= (1.0 + percent)
if __name__ == '__main__':
bob = Person('Bob Smith', 42, 30000, 'software')
sue = Person('Sue Jones', 45, 40000, 'hardware')
print(bob.name, sue.pay)
print(bob.lastName())
sue.giveRaise(.10)
print(sue.pay)

The output of this script is the same as the last, but the results
are being computed by methods now, not by hardcoded logic that appears
redundantly wherever it is required:

Bob Smith 40000
Smith
44000.0

One last
enhancement to our records before they become permanent:
because they are implemented as classes now, they naturally support
customization through the inheritance search mechanism in Python.
Example 1-16
, for instance, customizes
the last section’s
Person
class in
order to give a 10 percent bonus by default to managers whenever they
receive a raise (any relation to practice in the real world is purely
coincidental).

from person import Person
class Manager(Person):
def giveRaise(self, percent, bonus=0.1):
self.pay *= (1.0 + percent + bonus)
if __name__ == '__main__':
tom = Manager(name='Tom Doe', age=50, pay=50000)
print(tom.lastName())
tom.giveRaise(.20)
print(tom.pay)

When run, this script’s self-test prints the following:

Doe
65000.0

Here, the
Manager
class appears
in a module of its own, but it could have been added to the
person
module instead (Python doesn’t require
just one class per file). It inherits the constructor and last-name
methods from its superclass, but it customizes just the
giveRaise
method (there are a variety of ways
to code this extension, as we’ll see later). Because this change is
being added as a new subclass, the original
Person
class, and any objects generated from
it, will continue working unchanged. Bob and Sue, for example, inherit
the original raise logic, but Tom gets the custom version because of the
class from which he is created. In OOP, we program by
customizing
, not by changing.

In fact, code that uses our objects doesn’t need to be at all
aware of what the raise method does—it’s up to the object to do the
right thing based on the class from which it is created. As long as the
object supports the expected interface (here, a method called
giveRaise
), it will be compatible with the
calling code, regardless of its specific type, and even if its method
works differently than others.

If you’ve already studied Python, you may know this behavior as
polymorphism
; it’s a core property of the language,
and it accounts for much of your code’s flexibility. When the following
code calls the
giveRaise
method, for
example, what happens depends on the
obj
object being processed; Tom gets a 20
percent raise instead of 10 percent because of the
Manager
class’s customization:

>>>
from person import Person
>>>
from manager import Manager
>>>
bob = Person(name='Bob Smith', age=42, pay=10000)
>>>
sue = Person(name='Sue Jones', age=45, pay=20000)
>>>
tom = Manager(name='Tom Doe',  age=55, pay=30000)
>>>
db = [bob, sue, tom]
>>>
for obj in db:
obj.giveRaise(.10)
# default or custom
>>>
for obj in db:
print(obj.lastName(), '=>', obj.pay)
Smith => 11000.0
Jones => 22000.0
Doe => 36000.0

Before we
move on, there are a few coding alternatives worth noting
here. Most of these underscore the Python OOP model, and they serve as a
quick review.

class Manager(Person):
def giveRaise(self, percent, bonus=0.1):
Person.giveRaise(self, percent + bonus)

instance.method(arg1, arg2)
class.method(instance, arg1, arg2)

class Person:
def __str__(self):
return '<%s => %s>' % (self.__class__.__name__, self.name)
tom = Manager('Tom Jones', 50)
print(tom)                               # prints:  Tom Jones>

tom = Manager(name='Tom Doe', age=50, pay=50000, job='manager')

class Manager(Person):
def __init__(self, name, age, pay):
Person.__init__(self, name, age, pay, 'manager')

"""
Alternative implementation of person classes, with data, behavior,
and operator overloading (not used for objects stored persistently)
"""
class Person:
"""
a general person: data+logic
"""
def __init__(self, name, age, pay=0, job=None):
self.name = name
self.age  = age
self.pay  = pay
self.job  = job
def lastName(self):
return self.name.split()[-1]
def giveRaise(self, percent):
self.pay *= (1.0 + percent)
def __str__(self):
return ('<%s => %s: %s, %s>' %
(self.__class__.__name__, self.name, self.job, self.pay))
class Manager(Person):
"""
a person with custom raise
inherits general lastname, str
"""
def __init__(self, name, age, pay):
Person.__init__(self, name, age, pay, 'manager')
def giveRaise(self, percent, bonus=0.1):
Person.giveRaise(self, percent + bonus)
if __name__ == '__main__':
bob = Person('Bob Smith', 44)
sue = Person('Sue Jones', 47, 40000, 'hardware')
tom = Manager(name='Tom Doe', age=50, pay=50000)
print(sue, sue.pay, sue.lastName())
for obj in (bob, sue, tom):
obj.giveRaise(.10)                 # run this obj's giveRaise
print(obj)                         # run common __str__ method

 Sue Jones: hardware, 40000> 40000 Jones
 Bob Smith: None, 0.0>
 Sue Jones: hardware, 44000.0>
 Tom Doe: manager, 60000.0>