When I present
theftplibinterfaces
in Python classes, students often ask why programmers need to supply the
RETR string in the retrieval method. It’s a good
question—
the RETR string is the name of
the download command in the FTP protocol, butftplibis supposed to encapsulate that
protocol. As we’ll see in a moment, we have to supply an arguably
odd
STOR string for uploads as well. It’s boilerplate code
that you accept on faith once you see it, but that begs the question.
You could propose a patch toftplib,
but that’s not really a good answer for beginning Python students, and
it may break existing code (the interface is as it is for a
reason).
Perhaps a better answer is that Python makes it easy to extend the
standard library modules with higher-level interfaces of our own—with
just a few lines of reusable code, we can make the FTP interface look
any way we want in Python. For instance, we could, once and for all,
write utility modules that wrap theftplibinterfaces to hide the RETR string. If
we place these utility modules in a directory onPYTHONPATH, they become just as accessible asftplibitself, automatically reusable
in any Python script we write in the future. Besides removing the RETR
string requirement, a wrapper module could also make assumptions that
simplify FTP operations into single function calls.
For instance, given a module that encapsulates and simplifiesftplib, our Python fetch-and-play
script could be further reduced to the script shown in
Example 13-3
—essentially just two
function calls plus a password prompt, but with a net effect exactly
like
Example 13-1
when
run.
Example 13-3. PP4E\Internet\Ftp\getone-modular.py
#!/usr/local/bin/python
"""
A Python script to download and play a media file by FTP.
Uses getfile.py, a utility module which encapsulates FTP step.
"""
import getfile
from getpass import getpass
filename = 'monkeys.jpg'
# fetch with utility
getfile.getfile(file=filename,
site='ftp.rmi.net',
dir ='.',
user=('lutz', getpass('Pswd?')),
refetch=True)
# rest is the same
if input('Open file?') in ['Y', 'y']:
from PP4E.System.Media.playfile import playfile
playfile(filename)
Besides having a much smaller line count, the meat of this script
has been split off into a file for reuse elsewhere. If you ever need to
download a file again, simply import an existing function instead of
copying code with cut-and-paste editing. Changes in download operations
would need to be made in only one file, not everywhere we’ve copied
boilerplate code;getfile.getfilecould even be changed to useurllibrather thanftplibwithout affecting
any of its clients. It’s good engineering.
So just how would we go about writing such an FTP interface
wrapper (he asks, rhetorically)? Given theftpliblibrary module, wrapping downloads of
a particular file in a particular directory is straightforward.
Connected FTP objects support two download methods:
retrbinary
This method
downloads the requested file in binary mode,
sending its bytes in chunks to a supplied function, without
line-feed mapping. Typically, the supplied function is a write
method of an open local file object, such that the bytes are
placed in the local file on the client.
retrlines
This method
downloads the requested file in ASCII text mode,
sending each line of text to a supplied function with all
end-of-line characters stripped. Typically, the supplied
function adds a\nnewline
(mapped appropriately for the client machine), and writes the
line to a local file.
We will meet theretrlinesmethod in a later example; thegetfileutility module in
Example 13-4
always transfers in
binary mode withretrbinary. That
is, files are downloaded exactly as they were on the server, byte for
byte, with the server’s line-feed conventions in text files (you may
need to convert line feeds after downloads if they look odd in your
text editor—see your editor or system shell commands for pointers, or
write a Python script that opens and writes the text as
needed).
Example 13-4. PP4E\Internet\Ftp\getfile.py
#!/usr/local/bin/python
"""
Fetch an arbitrary file by FTP. Anonymous FTP unless you pass a
user=(name, pswd) tuple. Self-test FTPs a test file and site.
"""
from ftplib import FTP # socket-based FTP tools
from os.path import exists # file existence test
def getfile(file, site, dir, user=(), *, verbose=True, refetch=False):
"""
fetch a file by ftp from a site/directory
anonymous or real login, binary transfer
"""
if exists(file) and not refetch:
if verbose: print(file, 'already fetched')
else:
if verbose: print('Downloading', file)
local = open(file, 'wb') # local file of same name
try:
remote = FTP(site) # connect to FTP site
remote.login(*user) # anonymous=() or (name, pswd)
remote.cwd(dir)
remote.retrbinary('RETR ' + file, local.write, 1024)
remote.quit()
finally:
local.close() # close file no matter what
if verbose: print('Download done.') # caller handles exceptions
if __name__ == '__main__':
from getpass import getpass
file = 'monkeys.jpg'
dir = '.'
site = 'ftp.rmi.net'
user = ('lutz', getpass('Pswd?'))
getfile(file, site, dir, user)
This module is mostly just a repackaging of the FTP code we used
to fetch the image file earlier, to make it simpler and reusable.
Because it is a callable function, the exportedgetfile.getfilehere tries to be as robust
and generally useful as possible, but even a function this small
implies some design decisions. Here are a few usage notes:
Thegetfilefunction in this script runs in anonymous FTP mode
by default, but a two-item tuple containing a username and
password string may be passed to theuserargument in order to log in to
the remote server in nonanonymous mode. To use anonymous FTP,
either don’t pass theuserargument or pass it an empty tuple,(). The FTP objectloginmethod allows two optional
arguments to denote a username and password, and thefunction(*args)call syntax in
Example 13-4
sends it
whatever argument tuple you pass touseras individual arguments.
If passed, the last two arguments (verbose,refetch) allow us to turn off status
messages printed to thestdoutstream (perhaps undesirable in
a GUI context) and to force downloads to happen even if the file
already exists locally (the download overwrites the existing
local file).
These two arguments are coded as Python 3.X default
keyword-only arguments
, so if used they
must be passed by name, not position. Theuserargument instead can be passed
either way, if it is passed at all. Keyword-only arguments here
prevent passed verbose or refetch values from being incorrectly
matched against theuserargument if the user value is actually omitted in a call.
The caller is expected to handle exceptions; this function
wraps downloads in atry/finallystatement to guarantee that
the local output file is closed, but it lets exceptions
propagate. If used in a GUI or run from a thread, for instance,
exceptions may require special handling unknown in this
file.
If run standalone, this file downloads an image file again
from my website as a self-test (configure for your server and
file as desired), but the function will normally be passed FTP
filenames, site names, and directory names as well.
As in earlier examples, this script is careful to open the
local output file inwbbinary mode to suppress end-line mapping and conform to Python
3.X’s Unicode string model. As we learned in
Chapter 4
, it’s not impossible that
true binary datafiles may have bytes whose value is equal to a\nline-feed character;
opening inwtext mode
instead would make these bytes automatically expand to a\r\ntwo-byte sequence when
written locally on Windows. This is only an issue when run on
Windows; modewdoesn’t
change end-lines elsewhere.
As we also learned in
Chapter 4
, though, binary mode is
required to suppress the automatic
Unicode
translations
performed for text in Python 3.X.
Without binary mode, Python would attempt to encode fetched data
when written per a default or passed Unicode encoding scheme,
which might fail for some types of fetched text and would
normally fail for truly binary data such as images and
audio.
Becauseretrbinarywritesbytesstrings in 3.X,
we really cannot open the output file in text mode anyhow, orwritewill raise exceptions.
Recall that in 3.X text-mode files requirestrstrings, and binary mode files
expectbytes. Sinceretrbinarywritesbytesandretrlineswritesstr, they implicitly require binary
and text-mode output files, respectively. This constraint is
irrespective of end-line or Unicode issues, but it effectively
accomplishes those goals as well.
As we’ll see in later examples, text-mode retrievals have
additional encoding requirements; in fact,ftplibwill turn out to be a good
example of the impacts of Python 3.X’s Unicode string model on
real-world code. By always using binary mode in the script here,
we sidestep the issue altogether.
This function currently uses the same filename to identify
both the remote file and the local file where the download
should be stored. As such, it should be run in the directory
where you want the file to show up; useos.chdirto move to directories if
needed. (We could instead assume
filename
is the local file’s name, and strip the local directory withos.path.splitto get the
remote name, or accept two distinct filename arguments—local and
remote.)
Also notice that, despite its name, this module is very
different from the
getfile.py
script we studied
at the end of the sockets material in the preceding chapter. The
socket-basedgetfileimplemented
custom client and server-side logic to download a server file to a
client machine over raw sockets.
The newgetfilehere is a
client-side tool only. Instead of raw sockets, it uses the standard
FTP protocol to request a file from a server; all socket-level details
are hidden in the simplerftplibmodule’s implementation of the FTP client protocol. Furthermore, the
server here is a perpetually running program on the server machine,
which listens for and responds to FTP requests on a socket, on the
dedicated FTP port (number 21). The net functional effect is that this
script requires an FTP server to be running on the machine where the
desired file lives, but such a server is much more likely to be
available.
While we’re at it, let’s write a script to upload a single file
by FTP to a remote machine. The upload interfaces in the FTP module
are symmetric with the download interfaces. Given a connected FTP
object, its:
storbinarymethod
can be used to upload bytes from an open local file
object
storlinesmethod
can be used to upload text in ASCII mode from an
open local file object
Unlike the download interfaces, both of these methods are passed
a file object as a whole, not a file object method (or other
function). We will meet thestorlinesmethod in a later example. The
utility module in
Example 13-5
usesstorbinarysuch that the file whose name is
passed in is always uploaded verbatim—in binary mode, without Unicode
encodings or line-feed translations for the target machine’s
conventions. If this script uploads a text file, it will arrive
exactly as stored on the machine it came from, with client line-feed
markers and existing Unicode encoding.
Example 13-5. PP4E\Internet\Ftp\putfile.py
#!/usr/local/bin/python
"""
Store an arbitrary file by FTP in binary mode. Uses anonymous
ftp unless you pass in a user=(name, pswd) tuple of arguments.
"""
import ftplib # socket-based FTP tools
def putfile(file, site, dir, user=(), *, verbose=True):
"""
store a file by ftp to a site/directory
anonymous or real login, binary transfer
"""
if verbose: print('Uploading', file)
local = open(file, 'rb') # local file of same name
remote = ftplib.FTP(site) # connect to FTP site
remote.login(*user) # anonymous or real login
remote.cwd(dir)
remote.storbinary('STOR ' + file, local, 1024)
remote.quit()
local.close()
if verbose: print('Upload done.')
if __name__ == '__main__':
site = 'ftp.rmi.net'
dir = '.'
import sys, getpass
pswd = getpass.getpass(site + ' pswd?') # filename on cmdline
putfile(sys.argv[1], site, dir, user=('lutz', pswd)) # nonanonymous login
Notice that for portability, the local file is opened inrbbinary input mode this time to
suppress automatic line-feed character conversions. If this is binary
information, we don’t want any bytes that happen to have the value of
the\rcarriage-return character to
mysteriously go away during the transfer when run on a Windows client.
We also want to suppress Unicode encodings for nontext files, and we
want reads to produce thebytesstrings expected by thestorbinaryupload operation (more on input file modes later).
This script uploads a file you name on the command line as a
self-test, but you will normally pass in real remote filename, site
name, and directory name strings. Also like the download utility, you
may pass a(username, password)tuple to theuserargument to
trigger nonanonymous FTP mode (anonymous FTP is the default).