Programming Python (185 page)

In one form or another, processing text-based information is one of
the more common tasks that applications need to perform. This can include
anything from scanning a text file by columns to analyzing statements in a
language defined by a formal grammar. Such processing usually is
called
parsing
—analyzing the structure of a
text string. In this chapter, we’ll explore ways to handle language and
text-based information and summarize some Python development concepts in
sidebars along the way. In the process, we’ll meet string methods, text
pattern matching, XML and HTML parsers, and other tools.

Some of this material is advanced, but the examples are small to
keep this chapter short. For instance, recursive descent parsing is
illustrated with a simple example to show how it can be implemented in
Python. We’ll also see that it’s often unnecessary to write custom parsers
for each language processing task in Python. They can usually be replaced
by exporting APIs for use in Python programs, and sometimes by a single
built-in function call. Finally, this chapter closes by presenting
PyCalc—a calculator GUI written in Python, and the last major Python
coding example in this text. As we’ll see, writing calculators isn’t much
more difficult than juggling stacks while scanning text.

Expressions

Built-in string object expressions

Methods

Built-in string object method calls

Patterns

Regular expression pattern matching

Parsers: markup

XML and HTML text parsing

Parsers: grammars

Custom language parsers, both handcoded and generated

Embedding

Running Python code with
eval
and
exec
built-ins

And more

Natural language processing

Some readers may have come to this chapter seeking coverage of
Unicode text, too, but this topic is not presented here. For a look at
Python’s Unicode support, see
Chapter 2
’s
discussion of string tools,
Chapter 4
’s discussion of text and binary
file distinctions and encodings, and
Chapter 9
’s coverage of text in tkinter
GUIs. Unicode also appears in various Internet and database topics
throughout this book (e.g., email encodings).

Because Unicode is a core language topic, all these chapters will
also refer you to the fuller coverage of Unicode in
Learning
Python
, Fourth Edition. Most of the topics in this
chapter, including string methods and pattern matching, apply to Unicode
automatically simply because the Python 3.X
str
string type
is
Unicode, whether ASCII or wider.

str.find(
substr
)


Performs
substring searches

str.replace(
old
,
new
)


Performs
substring substitutions

str.split(
delimiter
)


Chops up a
string around a delimiter or whitespace

str.join(
iterable
)


Puts
substrings together with delimiters between

str.strip()


Removes l
eading and trailing whitespace

str.rstrip()


Removes
trailing whitespace only, if any

str.rjust(
width
)


Right-justifies
a string in a fixed-width field

str.upper()


Converts
to uppercase

str.isupper()


Tests whether
the string is uppercase

str.isdigit()


Tests
whether the string is all digit characters

str.endswith(
substr-or-tuple
)


Tests
for a substring (or a tuple of alternatives) at the
end

str.startswith(
substr-or-tuple
)


Tests for a
substring (or a tuple of alternatives) at the
front

By way of
review, let’s take a quick look at string methods in the
context of some of their most common use cases. As we saw when
generating HTML forwarding pages in
Chapter 6
, the string
replace
method is
often adequate by itself as a string
templating
tool—we can compute values and insert them at fixed positions in a
string with simple replacement calls:

>>>
template = '---$target1---$target2---'
>>>
val1 = 'Spam'
>>>
val2 = 'shrubbery'
>>>
template = template.replace('$target1', val1)
>>>
template = template.replace('$target2', val2)
>>>
template
'---Spam---shrubbery---'

As we also saw when generating HTML reply pages in the CGI scripts
of Chapters
15
and
16
, the
string
%
formatting operator is also
a powerful templating tool, especially when combined with
dictionaries—simply fill out a dictionary with values and apply multiple
substitutions to the HTML string all at once:

>>>
template = """
...
---
...
---%(key1)s---
...
---%(key2)s---
...
"""
>>>
>>>
vals = {}
>>>
vals['key1'] = 'Spam'
>>>
vals['key2'] = 'shrubbery'
>>>
print(template % vals)
---
---Spam---
---shrubbery---

Beginning with Python 2.4, the
string
module’s
Template
feature
is essentially a simplified and limited variation of the
dictionary-based format scheme just shown, but it allows some additional
call patterns which some may consider simpler:

>>>
vals
{'key2': 'shrubbery', 'key1': 'Spam'}
>>>
import string
>>>
template = string.Template('---$key1---$key2---')
>>>
template.substitute(vals)
'---Spam---shrubbery---'
>>>
template.substitute(key1='Brian', key2='Loretta')
'---Brian---Loretta---'

See the library manual for more on this extension. Although the
string datatype does not itself support the pattern-directed text
processing that we’ll meet later in this chapter, its tools are powerful
enough for many tasks.

In terms of this
chapter’s main focus, Python’s built-in tools for
splitting and joining strings around tokens turn out to be especially
useful when it comes to parsing text:

These two are among the most powerful of string methods. As we saw
in
Chapter 2
,
split
chops a string into a list of substrings
and
join
puts them back
together:

>>>
'A B C D'.split()
['A', 'B', 'C', 'D']
>>>
'A+B+C+D'.split('+')
['A', 'B', 'C', 'D']
>>>
'--'.join(['a', 'b', 'c'])
'a--b--c'

Despite their simplicity, they can handle surprisingly complex
text-parsing tasks. Moreover, string method calls are very fast because
they are implemented in C language code. For instance, to quickly
replace all tabs in a file with four periods, pipe the file into a
script that looks like this:

from sys import *
stdout.write(('.' * 4).join(stdin.read().split('\t')))

The
split
call here divides
input around tabs, and the
join
puts
it back together with periods where tabs had been. In this case, the
combination of the two calls is equivalent to using the simpler global
replacement string method call as follows:

stdout.write(stdin.read().replace('\t', '.' * 4))

As we’ll see in the next section, splitting strings is sufficient
for many text-parsing goals.

Let’s look next at
some practical applications of string splits and joins. In
many domains, scanning files by columns is a fairly common task. For
instance, suppose you have a file containing columns of numbers output
by another system, and you need to sum each column’s numbers. In Python,
string splitting is the core operation behind solving this problem, as
demonstrated by
Example 19-1
. As
an added bonus, it’s easy to make the solution a reusable tool in Python
by packaging it as an importable function.

#!/usr/local/bin/python
def summer(numCols, fileName):
sums = [0] * numCols                             # make list of zeros
for line in open(fileName):                      # scan file's lines
cols = line.split()                          # split up columns
for i in range(numCols):                     # around blanks/tabs
sums[i] += eval(cols[i])                 # add numbers to sums
return sums
if __name__ == '__main__':
import sys
print(summer(eval(sys.argv[1]), sys.argv[2]))    # '% summer.py cols file'

Notice that we use file iterators here to read line by line,
instead of calling the file
readlines
method
explicitly (recall from
Chapter 4
that
iterators avoid loading the entire file into memory all at once). The
file itself is a temporary object, which will be automatically closed
when garbage collected.

As usual for properly architected scripts, you
can both
import
this module and call
its function, and
run
it as a shell tool from the
command line. The
summer.py
script calls
split
to make a list of strings representing
the line’s columns, and
eval
to
convert column strings to numbers. Here’s an input file that uses both
blanks and tabs to separate columns, and the result of turning our
script loose on it:

C:\...\PP4E\Lang>
type table1.txt
1       5       10    2   1.0
2       10      20    4   2.0
3       15      30    8    3
4       20      40   16   4.0
C:\...\PP4E\Lang>
python summer.py 5 table1.txt
[10, 50, 100, 30, 10.0]

Also notice that because the summer script uses
eval
to convert file text to numbers, you
could really store arbitrary Python expressions in the file. Here, for
example, it’s run on a file of Python code snippets:

C:\...\PP4E\Lang>
type table2.txt
2     1+1          1<<1           eval("2")
16    2*2*2*2      pow(2,4)       16.0
3     len('abc')   [1,2,3][2]     {'spam':3}['spam']
C:\...\PP4E\Lang>
python summer.py 4 table2.txt
[21, 21, 21, 21.0]

def summer(numCols, fileName):
sums = [0] * numCols
for line in open(fileName):                     # use file iterators
cols = line.split(',')                      # assume comma-delimited
nums = [int(x) for x in cols]               # use limited converter
both = zip(sums, nums)                      # avoid nested for loop
sums = [x + y for (x, y) in both]           # 3.X: zip is an iterable
return sums

C:\...\PP4E\Lang>
type table3.txt
1,5,10,2,1
2,10,20,4,2
3,15,30,8,3
4,20,40,16,4
C:\...\PP4E\Lang>
python summer2.py 5 table3.txt
[10, 50, 100, 30, 10]

C:\...\PP4E\Lang>
python
>>>
print(open('table4.txt').read())
001.1 002.2 003.3
010.1 020.2 030.3 040.4
100.1 200.2 300.3

>>>
sums = {}
>>>
for line in open('table4.txt'):
...
cols = [float(col) for col in line.split()]
...
for pos, val in enumerate(cols):
...
sums[pos] = sums.get(pos, 0.0) + val
...
>>>
for key in sorted(sums):
...
print(key, '=', sums[key])
...
0 = 111.3
1 = 222.6
2 = 333.9
3 = 40.4
>>>
sums
{0: 111.3, 1: 222.6, 2: 333.90000000000003, 3: 40.4}