#!/usr/bin/python
"""
Same, but for easier maintenance, use HTML template strings, get the
Language table and input key from common module file, and get reusable
form field mockup utilities module for testing.
"""
import cgi, sys
from formMockup import FieldMockup                   # input field simulator
from languages2common import hellos, inputkey        # get common table, name
debugme = False
hdrhtml = """Content-type: text/html\n

Syntax
"""
langhtml = """
%s

%s

"""
def showHello(form):                                 # HTML for one language
choice = form[inputkey].value                    # escape lang name too
try:
print(langhtml % (cgi.escape(choice),
cgi.escape(hellos[choice])))
except KeyError:
print(langhtml % (cgi.escape(choice),
"Sorry--I don't know that language"))
def main():
if debugme:
form = {inputkey: FieldMockup(sys.argv[1])}  # name on cmd line
else:
form = cgi.FieldStorage()                    # parse real inputs
print(hdrhtml)
if not inputkey in form or form[inputkey].value == 'All':
for lang in hellos.keys():
mock = {inputkey: FieldMockup(lang)}     # not dict(n=v) here!
showHello(mock)
else:
showHello(form)
print('
')
if __name__ == '__main__': main()

Notice that in the prior section, although it’s wrong to embed an
unescaped
<
in the HTML code
reply, it’s perfectly all right to include it literally in the URL
string used to trigger the reply. In fact, HTML and URLs define
completely different characters as special. For instance, although
&
must be escaped as
&
inside HTML code, we have to use
other escaping schemes to code a literal
&
within a URL string (where it normally
separates parameters). To pass a language name like
a&b
to our script, we have to type the
following URL:

http://localhost/cgi-bin/languages2reply.py?language=a%26b

Here,
%26
represents
&
—the
&
is replaced with a
%
followed by the hexadecimal value (0x26) of
its ASCII code value (38). Similarly, as we suggested at the end of
Chapter 13
, to name C++ as a query
parameter in an explicit URL,
+
must
be escaped as
%2b
:

http://localhost/cgi-bin/languages2reply.py?language=C%2b%2b

Sending
C++
unescaped will not
work, because
+
is special in URL
syntax—it represents a space. By URL standards, most nonalphanumeric
characters are supposed to be translated to such escape sequences, and
spaces are replaced by
+
signs.
Technically, this convention is known as the
application/x-www-form-urlencoded
query string
format, and it’s part of the magic behind those bizarre URLs you often
see at the top of your browser as you surf the Web.

If you’re like me, you probably don’t have the hexadecimal value
of the ASCII code for
&
committed
to memory (though Python’s
hex(ord(c))
can help). Luckily, Python
provides tools that automatically implement URL escapes, just as
cgi.escape
does for HTML escapes. The
main thing to keep in mind is that HTML code and URL strings are written
with entirely different syntax, and so employ distinct escaping
conventions. Web users don’t generally care, unless they need to type
complex URLs explicitly—browsers handle most escape code details
internally. But if you write scripts that must generate HTML or URLs,
you need to be careful to escape characters that are reserved in either
syntax.

Because HTML and URLs have different syntaxes, Python provides two
distinct sets of tools for escaping their text. In the standard Python
library:

The
urllib.parse
module
also has tools for undoing URL escapes (
unquote
,
unquote_plus
), but HTML escapes are undone
during HTML parsing at large (e.g., by Python’s
html.parser
module). To illustrate the two
escape conventions and tools, let’s apply each tool set to a few simple
examples.

As we saw earlier,
cgi.escape
translates code for inclusion within HTML. We normally call this utility
from a CGI script, but it’s just as easy to explore its behavior
interactively:

>>>
import cgi
>>>
cgi.escape('a < b > c & d "spam"', 1)
'a < b > c & d "spam"'
>>>
s = cgi.escape("1<2 hello")
>>>
s
'1<2 <b>hello</b>'

Python’s
cgi
module
automatically converts characters that are special in HTML syntax
according to the HTML convention. It translates
<
,
>
,
and
&
with an extra true
argument,
"
, into escape sequences of
the form
&X;
, where the
X
is a mnemonic that denotes the original
character. For instance,
<
stands for the “less than” operator (
<
) and
&
denotes a literal ampersand
(
&
).

There is no
un
escaping tool in the CGI
module, because HTML escape code sequences are recognized within the
context of an HTML parser, like the one used by your web browser when a
page is downloaded. Python comes with a full HTML parser, too, in the
form of the standard module
html.parser
. We won’t go into details on the
HTML parsing tools here (they’re covered in
Chapter 19
in conjunction with text processing), but
to illustrate how escape codes are eventually undone, here is the HTML
parser module at work reading back the preceding output:

>>>
import cgi, html.parser
>>>
s = cgi.escape("1<2 hello")
>>>
s
'1<2 <b>hello</b>'
>>>
>>>
html.parser.HTMLParser().unescape(s)
'1<2 hello'

This uses a utility method on the HTML parser class to unquote. In
Chapter 19
, we’ll see that using this class
for more substantial work involves subclassing to override methods run
as callbacks during the parse upon detection of tags, data, entities,
and more. For more on full-blown HTML parsing, watch for the rest of
this story in
Chapter 19
.

By contrast, URLs reserve other characters as special and must
adhere to different escape conventions. As a result, we use different
Python library tools to escape URLs for transmission. Python’s
urllib.parse
module provides two tools that do
the translation work for us:
quote
,
which implements the standard
%XX
hexadecimal URL escape code sequences for most nonalphanumeric
characters, and
quote_plus
, which
additionally translates spaces to
+
signs. The
urllib.parse
module also
provides functions for unescaping quoted characters in a URL string:
unquote
undoes
%XX
escapes, and
unquote_plus
also changes plus signs back to
spaces. Here is the module at work, at the interactive prompt:

>>>
import urllib.parse
>>>
urllib.parse.quote("a & b #! c")
'a%20%26%20b%20%23%21%20c'
>>>
urllib.parse.quote_plus("C:\stuff\spam.txt")
'C%3A%5Cstuff%5Cspam.txt'
>>>
x = urllib.parse.quote_plus("a & b #! c")
>>>
x
'a+%26+b+%23%21+c'
>>>
urllib.parse.unquote_plus(x)
'a & b #! c'

URL escape sequences embed the hexadecimal values of nonsafe
characters following a
%
sign (this
is usually their ASCII codes). In
urllib.parse
, nonsafe characters are usually
taken to include everything except letters, digits, and a handful of
safe special characters (any in
'_.-'
), but the two tools differ on forward
slashes, and you can extend the set of safe characters by passing an
extra string argument to the quote calls to customize the
translations:

>>>
urllib.parse.quote_plus("uploads/index.txt")
'uploads%2Findex.txt'
>>>
urllib.parse.quote("uploads/index.txt")
'uploads/index.txt'
>>>
>>>
urllib.parse.quote_plus("uploads/index.txt", '/')
'uploads/index.txt'
>>>
urllib.parse.quote("uploads/index.txt", '/')
'uploads/index.txt'
>>>
urllib.parse.quote("uploads/index.txt", '')
'uploads%2Findex.txt'
>>>
>>>
urllib.parse.quote_plus("uploads\index.txt")
'uploads%5Cindex.txt'
>>>
urllib.parse.quote("uploads\index.txt")
'uploads%5Cindex.txt'
>>>
urllib.parse.quote_plus("uploads\index.txt", '\\')
'uploads\\index.txt'

Note that Python’s
cgi
module
also translates URL escape sequences back to their original characters
and changes
+
signs to spaces during
the process of extracting input information. Internally,
cgi.FieldStorage
automatically calls
urllib.parse
tools which unquote if needed to
parse and unescape parameters passed at the end of URLs. The upshot is
that CGI scripts get back the original, unescaped URL strings, and don’t
need to unquote values on their own. As we’ve seen, CGI scripts don’t
even need to know that inputs came from a URL at all.

We’ve seen how to escape text inserted into both HTML and URLs.
But what do we do for URLs inside HTML? That is, how do we escape when
we generate and embed text inside a URL, which is itself embedded inside
generated HTML code? Some of our earlier examples used hardcoded URLs
with appended input parameters inside

hyperlink tags; the file
languages2.py
, for instance, prints HTML that
includes a URL:

Because the URL here is embedded in HTML, it must at least be
escaped according to HTML conventions (e.g., any
<
characters must become
<
), and any spaces should be translated
to
+
signs per URL conventions. A
cgi.escape(url)
call followed by the
string
url.replace(" ", "+")
would
take us this far, and would probably suffice for most cases.

That approach is not quite enough in general, though, because HTML
escaping conventions are not the same as URL conventions. To robustly
escape URLs embedded in HTML code, you should instead call
urllib.parse.quote_plus
on the URL string, or
at least most of its components, before adding it to the HTML text. The
escaped result also satisfies HTML escape conventions, because
urllib.parse
translates more characters than
cgi.escape
, and the
%
in URL escapes is not special to
HTML.

hello

file.py?name=a&job=b&=cS=d<=e
result in IE
file.py?name=a&job=b&=c%A7=d%3C=e
result in Chrome (0x3C is <)

>>>
from html.parser import HTMLParser
>>>
html = open('test-escapes.html').read()
>>>
HTMLParser().unescape(html)
'\nhello\n'

hello

file.py?name=a&job=b&=c§=d<=e
result in both IE and Chrome
>>>
h = 'hello'
>>>
HTMLParser().unescape(h)
'hello'

file.py?name=a;job=b;amp=c;sect=d;lt=e

"cgi-bin/badlink.py?name=a&job=b&=c§=d<=e">unescaped
"cgi-bin/badlink.py?name=a&job=b&amp=c&sect=d&lt=e">escaped
"cgi-bin/badlink.py?name=a;job=b;amp=c;sect=d;lt=e">alternative

import cgi, sys
form = cgi.FieldStorage()      # print all inputs to stderr; stodout=reply page
for name in form.keys():
print('[%s:%s]' % (name, form[name].value), end=' ', file=sys.stderr)

mark-VAIO - - [16/Jun/2010 10:43:24] b'[:c\xa7=d<=e] [job:b] [name:a] '
mark-VAIO - - [16/Jun/2010 10:43:24] CGI script exited OK
mark-VAIO - - [16/Jun/2010 10:43:27] b'[amp:c] [job:b] [lt:e] [name:a] [sect:d]'
mark-VAIO - - [16/Jun/2010 10:43:27] CGI script exited OK
mark-VAIO - - [16/Jun/2010 10:43:30] b'[amp:c] [job:b] [lt:e] [name:a] [sect:d]'
mark-VAIO - - [16/Jun/2010 10:43:30] CGI script exited OK

>>>
link = 'file.py?name=a&job=b&=c§=d<=e'
# escape for HTML
>>>
import cgi
>>>
cgi.escape(link)
'file.py?name=a&job=b&amp=c&sect=d&lt=e'
# escape for URL
>>>
import urllib.parse
>>>
elink = urllib.parse.quote_plus(link)
>>>
elink
'file.py%3Fname%3Da%26job%3Db%26amp%3Dc%26sect%3Dd%26lt%3De'
# URL satisfies HTML too: same
>>>
cgi.escape(elink
)
'file.py%3Fname%3Da%26job%3Db%26amp%3Dc%26sect%3Dd%26lt%3De'