Programming Python (162 page)

This chapter is the fifth in our survey of Python Internet
programming, and it continues
Chapter 15
’s
discussion. There, we explored the fundamentals of server-side Common
Gateway Interface (CGI) scripting in Python. Armed with that knowledge,
this chapter moves on to a larger case study that underscores advanced CGI
and server-side web scripting topics.

This chapter presents
PyMailCGI—a “webmail” website for reading and sending email
that illustrates security concepts, hidden form fields, URL generation,
and more. Because this system is similar in spirit to the PyMailGUI
program we studied in
Chapter 14
, this
example also serves as a comparison of the web and nonweb application
models. This case study is founded on basic CGI scripting, but it
implements a complete website that does something more useful than
Chapter 15
’s examples.

As usual in this book, this chapter splits its focus between
application-level details and Python programming concepts. For instance,
because this is a fairly large case study, it illustrates system design
concepts that are important in actual projects. It also says more about
CGI scripts in general: PyMailCGI expands on the notions of state
retention and security concerns and encryption.

The system presented here is neither particularly flashy nor feature
rich as websites go (in fact, the initial cut of PyMailCGI was thrown
together during a layover at Chicago’s O’Hare airport). Alas, you will
find neither dancing bears nor blinking lights at this site. On the other
hand, it was written to serve a real purpose, speaks more to us about CGI
scripting, and hints at just how far Python server-side programs can take
us. As outlined at the start of this part of the book, there are
higher-level frameworks, systems, and tools that build upon ideas we will
apply here. For now, let’s have some fun with Python on the Web.

At the top level,
PyMailCGI allows users to view incoming email with the
Post Office Protocol (POP) interface and to send new mail by Simple Mail
Transfer Protocol (SMTP). Users also have the option of replying to,
forwarding, or deleting an incoming email while viewing it. As
implemented, anyone can send email from a PyMailCGI site, but to view
your email, you generally have to install PyMailCGI on your own computer
or web server account, with your own mail server information (due to
security concerns described later).

Viewing and sending email sounds simple enough, and we’ve already
coded this a few times in this book. But the required interaction
involves a number of distinct web pages, each requiring a CGI script or
HTML file of its own. In fact, PyMailCGI is a fairly
linear
system—in the most complex user interaction
scenario, there are six states (and hence six web pages) from start to
finish. Because each page is usually generated by a distinct file in the
CGI world, that also implies six source files.

Technically, PyMailCGI could also be described as a
state machine
, though very little
state is transferred from state to state. Scripts pass user and message
information to the next script in hidden form fields and query
parameters, but there are no client-side cookies or server-side
databases in the current version. Still, along the way we’ll encounter
situations where more advanced state retention tools could be an
advantage.

To help keep track of how all of PyMailCGI’s source files fit into
the overall system, I jotted down the file in
Example 16-1
before starting any real
programming. It informally sketches the user’s flow through the system
and the files invoked along the way. You can certainly use more formal
notations to describe the flow of control and information through states
such as web pages (e.g., dataflow diagrams), but for this simple
example, this file gets the job done.

file or script                           creates
--------------                           -------
[pymailcgi.html]                         Root window
=> [onRootViewLink.py]                  Pop password window
=> [onViewPswdSubmit.py]            List window (loads all pop mail)
=> [onViewListLink.py]          View Window + pick=del|reply|fwd (fetch)
=> [onViewPageAction.py]    Edit window, or delete+confirm (del)
=> [onEditPageSend.py]  Confirmation (sends smtp mail)
=> back to root
=> [onRootSendLink.py]                  Edit Window
=> [onEditPageSend.py]              Confirmation (sends smtp mail)
=> back to root

This file simply lists all the source files in the system, using
=>
and indentation to denote the
scripts they trigger.

For instance, links on the
pymailcgi.html
root page
invoke
onRootViewLink.py
and
onRootSendLink.py
, both executable scripts. The
script
onRootViewLink.py
generates a password page,
whose Submit button in turn triggers
onViewPswdSubmit.py
, and so on. Notice that both
the view and the send actions can wind up triggering
onEdit
Page
Send.py
to send a new mail; view
operations get there after the user chooses to reply to or forward an
incoming mail.

In a system such as this, CGI scripts make little sense in
isolation, so it’s a good idea to keep the overall page flow in mind;
refer to this file if you get lost. For additional context,
Figure 16-1
shows the overall contents of this site,
viewed as directory listings under Cygwin on Windows in a shell
window.

Figure 16-1. PyMailCGI contents

When you install this site, all the files you see here are
uploaded to a
PyMailCgi
subdirectory of your web
directory on your server’s machine. Besides the page-flow HTML and CGI
script files invoked by user interaction, PyMailCGI uses a handful of
utility modules:

PyMailCGI also reuses parts of the
mailtools
module
package and
mailconfig.py
module we wrote in
Chapter 13
. The former of these is accessible to
imports from
the
PP4E
package root
, and the latter is largely copied by a local
version in the
PyMailCgi
directory so that it can differ between PyMailGUI and PyMailCGI. The
externs.py
module is intended to hide
these modules’ actual locations, in case
the install structure varies
on some
machines.

In fact, this system again demonstrates the powers of
code reuse
in a practical way. In this edition, it
gets a great deal of logic for free from the new
mailtools
package of
Chapter 13
—message loading, sending, deleting,
parsing, composing, decoding and encoding, and attachments—even though
that package’s modules were originally developed for the PyMailGUI
program. When it came time to update PyMailCGI later, tools for handling
complex things such as attachments and message text searches were
already in place. See
Chapter 13
for
mailtools
source code.

As usual, PyMailCGI also uses a variety of standard library
modules:
smtplib
,
poplib
,
email.*
,
cgi
,
urllib.*
, and the like. Thanks to the reuse of
both custom and standard library code, this system achieves much in a
minimal amount of code. All told, PyMailCGI consists of just 846 lines
of new code, including whitespace, comments, and the top-level HTML file
(see file
linecounts.xls
in this
system’s source directory for details; the prior edition’s version
claimed to be some 835 new lines).

This compares favorably to the size of the PyMailGUI client-side
“desktop” program in
Chapter 14
, but most
of this difference owes to the vastly more limited functionality in
PyMailCGI—there are no local save files, no transfer thread overlap, no
message caching, no inbox synchronization tests or recovery, no
multiple-message selections, no raw mail text views, and so on.
Moreover, as the next section describes, PyMailCGI’s Unicode
policies are substantially more limited in this release,
and although arbitrary attachments can be viewed, sending binary and
some text attachments is not supported in the current version because of
a Python 3.1 issue.

In other words, PyMailCGI is really something of a
prototype
, designed to illustrate web scripting and
system design concepts in this book, and serve as a springboard for
future work. As is, it’s nowhere near as far along the software
evolutionary scale as PyMailGUI. Still, we’ll see that PyMailCGI’s code
factoring and reuse of existing modules allow it to implement much in a
surprisingly small amount of
code.

In this fourth edition,
PyMailCGI has been ported to run under Python 3.X. In
addition, this version inherits and employs a variety of new features
from the
mailtools
module, including
mail header decoding and encoding, main mail text encoding, the ability
to limit mail headers fetched, and more. Notably, there is new support
for Unicode and Internationalized character sets as follows:

Note that this version relies upon web browsers’ ability to
display arbitrary kinds of Unicode text. It does not emit any sort of
“meta” tag to declare encodings in the HTML reply pages generated for
mail view and composition. For instance, a properly formed HTML document
can often declare its encoding this way:

Such headers are omitted here. This is in part due to the fact
that the mail might have arbitrary and even mixed types of text among is
message and headers, which might also clash with encoding in the HTML of
the reply itself. Consider a mail index list page that displays headers
of multiple mails; because each mail’s Subject and From might be
encoding in a different character set (one Russian, one Chinese, and so
on), a single encoding declaration won’t suffice (though UTF-8’s
generality can often come to the rescue). Resolving such mixed character
set cases is left to the browser, which may ultimately require
assistance from the user in the form of encoding choices. Such displays
work in PyMailGUI because we pass decoded Unicode text to the tkinter
Text widget, which handles arbitrary Unicode code points well. In
PyMailCGI, we’re largely finessing this issue to keep this example
short.

Moreover, both text and binary attachments of fetched mails are
simply saved in binary form and opened by filename in browsers when
their links are clicked, relying again on browsers to do the right
thing. Text attachments for sends are also subject to the CGI upload
limitations described in the note just ahead. Beyond all this, Python
3.1 appears to have an issue printing some types of Unicode text to the
standard output stream in the CGI context, which necessitates a
workaround in the main utilities module here that opens
stdout
in binary mode and writes text as
encoded bytes (see the code for more details).

This Unicode/i18n
support is substantially less rich than that in PyMailGUI.
However, given that we can’t prompt for encodings here, and given that
this book is running short on time and space in general, improving this
for cases and browsers where it might matter is left as a suggested
exercise.

For more on specific 3.0 fourth-edition changes made, see the
comments marked with “3.0” in the program code files listed ahead. In
addition, all the features added for the prior edition are still here,
as described in the next
section.