"""
fetch a file from an HTTP (web) server over sockets via urllib;  urllib supports
HTTP, FTP, files, and HTTPS via URL address strings;  for HTTP, the URL can name
a file or trigger a remote CGI script;  see also the urllib example in the FTP
section, and the CGI script invocation in a later chapter;  files can be fetched
over the net with Python in many ways that vary in code and server requirements:
over sockets, FTP, HTTP, urllib, and CGI outputs;  caveat: should run filename
through urllib.parse.quote to escape properly unless hardcoded--see later chapters;
"""
import sys
from urllib.request import urlopen
showlines = 6
try:
servername, filename = sys.argv[1:]              # cmdline args?
except:
servername, filename = 'learning-python.com', '/index.html'
remoteaddr = 'http://%s%s' % (servername, filename)  # can name a CGI script too
print(remoteaddr)
remotefile = urlopen(remoteaddr)                     # returns input file object
remotedata = remotefile.readlines()                  # read data directly here
remotefile.close()
for line in remotedata[:showlines]: print(line)      # bytes with embedded \n

Example 13-31. PP4E\Internet\Other\http-getfile-urllib2.py

"""
fetch a file from an HTTP (web) server over sockets via urlllib;  this version
uses an interface that saves the fetched data to a local binary-mode file; the
local filename is either passed in as a cmdline arg or stripped from the URL with
urllib.parse: the filename argument may have a directory path at the front and query
parameters at end, so os.path.split is not enough (only splits off directory path);
caveat: should urllib.parse.quote filename unless known ok--see later chapters;
"""
import sys, os, urllib.request, urllib.parse
showlines = 6
try:
servername, filename = sys.argv[1:3]              # first 2 cmdline args?
except:
servername, filename = 'learning-python.com', '/index.html'
remoteaddr = 'http://%s%s' % (servername, filename)   # any address on the Net
if len(sys.argv) == 4:                                # get result filename
localname = sys.argv[3]
else:
(scheme, server, path, parms, query, frag) = urllib.parse.urlparse(remoteaddr)
localname = os.path.split(path)[1]
print(remoteaddr, localname)
urllib.request.urlretrieve(remoteaddr, localname)       # can be file or script
remotedata = open(localname, 'rb').readlines()          # saved to local file
for line in remotedata[:showlines]: print(line)         # file is bytes/binary

The next listing shows this script being used to trigger a
remote program. As before, if you don’t give the local filename
explicitly, the script strips the base filename out of the filename
argument. That’s not always easy or appropriate for program
invocations—
the filename can contain
both a remote directory path at the front and query parameters at the
end for a remote program invocation.

Given a script invocation URL and no explicit output filename,
the script extracts the base filename in the middle by using first the
standard
urllib.parse
module to pull out the file path, and then
os.path.split
to strip off the directory
path. However, the resulting filename is a remote script’s name, and
it may or may not be an appropriate place to store the data locally.
In the first run that follows, for example, the script’s output goes
in a local file called
languages.py
, the script
name in the middle of the URL; in the second, we instead name the
output
CxxSyntax.html
explicitly to suppress
filename extraction:

C:\...\PP4E\Internet\Other>
python http-getfile-urllib2.py localhost
/cgi-bin/languages.py?language=Scheme
http://localhost/cgi-bin/languages.py?language=Scheme languages.py
b'\n'
b'
Syntax
\n'
b'
Scheme
\n'
b' (display "Hello World") (newline)  \n'
b'

\n'
b'
\n'
C:\...\PP4E\Internet\Other>
python http-getfile-urllib2.py localhost
/cgi-bin/languages.py?language=C++ CxxSyntax.html
http://localhost/cgi-bin/languages.py?language=C++ CxxSyntax.html
b'\n'
b'
Syntax
\n'
b'
C  
\n'
b"Sorry--I don't know that language\n"
b'

\n'
b'
\n'

The remote script returns a not-found message when passed “C++”
in the last command here. It turns out that “+” is a special character
in URL strings (meaning a space), and to be robust, both of the
urllib
scripts we’ve just written
should really run the
filename
string through something called
urllib.parse.quote
, a tool that escapes
special characters for transmission. We will talk about this in depth
in
Chapter 15
, so consider this a
preview for now. But to make this invocation work, we need to use
special sequences in the constructed URL. Here’s how to do it by
hand:

C:\...\PP4E\Internet\Other>
python http-getfile-urllib2.py  localhost
/cgi-bin/languages.py?language=C%2b%2b CxxSyntax.html
http://localhost/cgi-bin/languages.py?language=C%2b%2b CxxSyntax.html
b'\n'
b'
Syntax
\n'
b'
C++
\n'
b' cout << "Hello World" << endl;     \n'
b'

\n'
b'
\n'

The odd
%2b
strings in this
command line are not entirely magical: the escaping required for URLs
can be seen by running standard Python tools manually—this is what
these scripts should do automatically to be able to handle all
possible cases well;
url
lib
.
parse
.
unquote
can undo these escapes if
needed:

C:\...\PP4E\Internet\Other>
python
>>>
import urllib.parse
>>>
urllib.parse.quote('C++')
'c%2B%2B'

Again, don’t work too hard at understanding these last few
commands; we will revisit URLs and URL escapes in
Chapter 15
, while exploring server-side
scripting in Python. I will also explain there why the C++ result came
back with other oddities like
<<
—
HTML escapes for
<<
, generated by the tool
cgi.escape
in the script on the
server that produces the reply, and usually undone by HTML parsers
including Python’s
html.parser
module we’ll meet in
Chapter 19
:

>>>
import cgi
>>>
cgi.escape('<<')
'<<'

Also in
Chapter 15
, we’ll meet
urllib
support for
proxies
, and its support for client-side
cookies
. We’ll discuss the related HTTPS concept
in
Chapter 16
—HTTP transmissions over
secure sockets, supported by
urllib.request
on the client side if SSL
support is compiled into your Python. For now, it’s time to wrap up
our look at the Web, and the Internet at large, from the client side
of the
fence.