\chapter{File and Directory Access in Python}
\label{chap:filedir}

Lots of Python applications involve files and directories.  This
chapter shows you the basics.

\section{Files}

\subsection{Some Basic File Operations}
\label{basic}

A list of Python file operations can be obtained through the Python
{\bf dir()} command:

\begin{Verbatim}[fontsize=\relsize{-2}]
>>> dir(file)
['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__',
'__init__', '__iter__', '__new__', '__reduce__', '__reduce_ex__',
'__repr__', '__setattr__', '__str__', 'close', 'closed', 'encoding',
'fileno', 'flush', 'isatty', 'mode', 'name', 'newlines', 'next', 'read',
'readinto', 'readline', 'readlines', 'seek', 'softspace', 'tell',
'truncate', 'write', 'writelines', 'xreadlines']
\end{Verbatim}

Following is an overview of many of those operations.  I am beginning in
a directory {\bf /a}, which has a file {\bf x}, consisting of

\begin{Verbatim}[fontsize=\relsize{-2}]
a
bc
def
\end{Verbatim}

a file {\bf y}, consisting of

\begin{Verbatim}[fontsize=\relsize{-2}]
uuu
vvv
\end{Verbatim}

and a subdirectory {\bf b}, which is empty.

\begin{Verbatim}[fontsize=\relsize{-2},numbers=left]
>>> f = open('x')  # open, read-only (default)
>>> f.name  # recover name from file object
'x'
>>> f.readlines()
['a\n', 'bc\n', 'def\n']
>>> f.readlines()  # already at end of file, so this gets nothing
[]
>>> f.seek(0)  # go back to byte 0 of the file
>>> f.readlines()
['a\n', 'bc\n', 'def\n']
>>> f.seek(2)  # go to byte 2 in the file
>>> f.readlines()
['bc\n', 'def\n']
>>> f.seek(0)
>>> f.read()  # read to EOF, returning bytes in a string
'a\nbc\ndef\n'
>>> f.seek(0)
>>> f.read(3)  # read only 3 bytes this time
'a\nb'
\end{Verbatim}

If you need to read just one line, use {\bf readline()}.

You can use a file as an iterator:

\begin{Verbatim}[fontsize=\relsize{-2},numbers=left]
>>> f = open('x')
>>> i = 0
>>> for l in f:
...    print 'line %d:' % i,l[:-1]
...    i += 1
...
line 0: a
line 1: bc
line 2: def
\end{Verbatim}

Note, by the way, how we stripped off the newline character (which we
wanted to do because {\bf print} would add one and we don't want two) by
using {\bf l[:-1]} instead of {\bf l}.

So, how do we write to files?

\begin{Verbatim}[fontsize=\relsize{-2},numbers=left]
>>> g = open('z','w')  # open for writing
>>> g.write('It is a nice sunny day')  # takes just one argument
>>> g.write(' today.\n')
>>> g.close()
>>> import os
>>> os.system('cat z')  # issue a shell command, which also gives an exit code
It is a nice sunny day today.
0
>>> g = open('gy','w')
>>> g.writelines(['Here is a line,\n','and another.\n'])
>>> os.system('cat gy')
0
>>> g.close()
>>> os.system('cat gy')
Here is a line,
and another.
0
>>> a = open('yy','w')
>>> a.write('abc\n')
>>> a.close()
>>> os.system('cat yy')
abc
0
>>> b = open('yy','a')  # open in append mode
>>> b.write('de\n')
>>> b.close()
>>> os.system('cat yy')
abc
de
0
\end{Verbatim}

As in Unix, {\bf stdin} and {\bf stdout} count as files too ({\it
file-like objects}, in Python parlance), so we can use the same
operations, e.g.: 

\begin{Verbatim}[fontsize=\relsize{-2},numbers=left]
>>> import sys
>>> w = sys.stdin.readlines()  # keyboard input, ctrl-d to end 
moon, sun
and
stars
>>> w
['moon, sun\n', 'and\n', 'stars\n']
\end{Verbatim}

\subsection{Seek Example}

Things in the real world are often of huge scale, e.g.\ datasets with
hundreds of millions of lines at Google, the Bank of America's crucial
dependence on optimizing the placement of various files on disk
(otherwise they would not be able to keep up with the workload, e.g.\
for check transactions), etc.

Their methods are of course extremely fine-tuned, but here is how we
could use \textbf{seek()} to improve the performance of frequently-read
text files.  That function enables us to go directly to a specific byte
in the file, without starting at the beginning --- thereby potentially
saving huge amounts of read time.

To exploit this, we could establish an index of byte positions within
the file, at which the lines start:

\begin{lstlisting}
def makeLineIdx(fname):
   f = open(fname)
   lns = f.readlines()
   lens = map(len,lns)
   idxs = [0]
   tot = 0
   for i in range(len(lns)-1):
     tot += lens[i]
     idxs.append(tot)
   return idxs

def midReadLines(fname,fnameIdx,startLine):
   f = open(fname)
   f.seek(fnameIdx[startLine])
   return f.readlines()
\end{lstlisting}

\section{Directories}

\subsection{Some Basic Directory Operations}

The {\bf os} module is huge. Here's a start to seeing how big it is:

\begin{Verbatim}[fontsize=\relsize{-2}]
>>> import os
>>> dir(os)
['EX_CANTCREAT', 'EX_CONFIG', 'EX_DATAERR', 'EX_IOERR', 'EX_NOHOST',
'EX_NOINPUT', 'EX_NOPERM', 'EX_NOUSER', 'EX_OK', 'EX_OSERR',
'EX_OSFILE', 'EX_PROTOCOL', 'EX_SOFTWARE', 'EX_TEMPFAIL',
'EX_UNAVAILABLE', 'EX_USAGE', 'F_OK', 'NGROUPS_MAX', 'O_APPEND',
'O_CREAT', 'O_DIRECT', 'O_DIRECTORY', 'O_DSYNC', 'O_EXCL',
'O_LARGEFILE', 'O_NDELAY', 'O_NOCTTY', 'O_NOFOLLOW', 'O_NONBLOCK',
'O_RDONLY', 'O_RDWR', 'O_RSYNC', 'O_SYNC', 'O_TRUNC', 'O_WRONLY',
'P_NOWAIT', 'P_NOWAITO', 'P_WAIT', 'R_OK', 'TMP_MAX', 'UserDict',
'WCOREDUMP', 'WEXITSTATUS', 'WIFEXITED', 'WIFSIGNALED', 'WIFSTOPPED',
'WNOHANG', 'WSTOPSIG', 'WTERMSIG', 'WUNTRACED', 'W_OK', 'X_OK',
'_Environ', '__all__', '__builtins__', '__doc__', '__file__',
'__name__', '_copy_reg', '_execvpe', '_exists', '_exit',
'_get_exports_list', '_make_stat_result', '_make_statvfs_result',
'_pickle_stat_result', '_pickle_statvfs_result', '_spawnvef',
'_urandomfd', 'abort', 'access', 'altsep', 'chdir', 'chmod', 'chown',
'chroot', 'close', 'confstr', 'confstr_names', 'ctermid', 'curdir',
'defpath', 'devnull', 'dup', 'dup2', 'environ', 'error', 'execl',
'execle', 'execlp', 'execlpe', 'execv', 'execve', 'execvp', 'execvpe',
'extsep', 'fchdir', 'fdatasync', 'fdopen', 'fork', 'forkpty',
'fpathconf', 'fstat', 'fstatvfs', 'fsync', 'ftruncate', 'getcwd',
'getcwdu', 'getegid', 'getenv', 'geteuid', 'getgid', 'getgroups',
'getloadavg', 'getlogin', 'getpgid', 'getpgrp', 'getpid', 'getppid',
'getsid', 'getuid', 'isatty', 'kill', 'killpg', 'lchown', 'linesep',
'link', 'listdir', 'lseek', 'lstat', 'major', 'makedev', 'makedirs',
'minor', 'mkdir', 'mkfifo', 'mknod', 'name', 'nice', 'open', 'openpty',
'pardir', 'path', 'pathconf', 'pathconf_names', 'pathsep', 'pipe',
'popen', 'popen2', 'popen3', 'popen4', 'putenv', 'read', 'readlink',
'remove', 'removedirs', 'rename', 'renames', 'rmdir', 'sep', 'setegid',
'seteuid', 'setgid', 'setgroups', 'setpgid', 'setpgrp', 'setregid',
'setreuid', 'setsid', 'setuid', 'spawnl', 'spawnle', 'spawnlp',
'spawnlpe', 'spawnv', 'spawnve', 'spawnvp', 'spawnvpe', 'stat',
'stat_float_times', 'stat_result', 'statvfs', 'statvfs_result',
'strerror', 'symlink', 'sys', 'sysconf', 'sysconf_names', 'system',
'tcgetpgrp', 'tcsetpgrp', 'tempnam', 'times', 'tmpfile', 'tmpnam',
'ttyname', 'umask', 'uname', 'unlink', 'unsetenv', 'urandom', 'utime',
'wait', 'waitpid', 'walk', 'write']
\end{Verbatim}

To see some examples, we start in the directory {\bf /a} as above.

\begin{Verbatim}[fontsize=\relsize{-2},numbers=left]
>>> os.path.curdir  # get current directory (Pythonic humor)
'.'
>>> t = os.path.abspath(os.path.curdir)  # get its full path
>>> t
'/a'
>>> os.path.basename(t)  # get the tail end of the path
'a'
>>> os.getcwd()  # another way to get the full path
'/a'
>>> cwd = os.getcwd()
>>> os.listdir(cwd)  # see what's in this directory (excl. '.','..')
['b', 'x', 'y', 'z']
>>> os.stat('x')  # get all the information about a file
(33152, 5079544L, 773L, 1, 0, 0, 9L, 1112979025, 1112977272, 1112977272)
>>> # the numbers above are: inode protection mode; inode number; device 
>>> # inode resides on; number of links to the inode; user id of the owner;
>>> # group id of the owner; size in bytes; time of last access; time 
>>> # of last modification; "ctime" as reported by OS
>>> os.path.getsize('x')  # get file size; argument must be string
9L
>>> os.path.isdir('x')  # is this file a directory?
False
>>> os.path.isdir('b')
True
>>> os.path.getmtime('x')  # time of last modification (seconds since Epoch)
1112977272
>>> import time
>>> time.ctime(os.path.getmtime('x'))  # translate that to English
'Fri Apr  8 09:21:12 2005'
>>> os.chdir('b')  # change to subdirectory b
>>> os.listdir('.')
[]
>>> os.mkdir('c')  # make a new directory
>>> os.listdir('.')
['c']
\end{Verbatim}

\subsection{Example:  Finding a File}

The function {\bf findfile()} searches for a file (which
could be a directory) in the specified directory tree, returning the
full path name of the first instance of the file found with the
specified name, or returning None if not found.

For instance, suppose we have the directory tree {\bf /a} shown in
Section \ref{basic}, except that {\bf /b} contains a file {\bf z}.  Then
the code

\begin{lstlisting}
print findfile('/a','y')
print findfile('/a','b')
print findfile('/a','u')
print findfile('/a','z')
print findfile('/a/b','z')
\end{lstlisting}

produces the output

\begin{lstlisting}
/a/y
/a/b
None
/a/b/z
/a/b/z
\end{lstlisting}

\begin{lstlisting}[numbers=left]
import os

# returns full path name of flname in the tree rooted at treeroot;
# returns None if not found; directories do count as finding the file
def findfile(treeroot,flname):
   os.chdir(treeroot)
   currfls = os.listdir('.')
   for fl in currfls:
      if fl == flname:
         return os.path.abspath(fl)
   for fl in currfls:
      if os.path.isdir(fl):
         tmp = findfile(fl,flname)
         if not tmp == None: return tmp
   return None

def main():
   print findfile('/a','y')
   print findfile('/a','u')
   print findfile('/a','z')
   print findfile('/a/b','z')
\end{lstlisting}

\subsection{The Powerful {\tt walk()} Function}

The function {\bf os.path.walk()} does a recursive descent down a directory
tree, stopping in each subdirectory to perform user-coded actions.
This is quite a powerful tool.\footnote{Unix users will recognize some
similarity to the Unix {\bf find} command.}  

The form of the call is 

\begin{Verbatim}[fontsize=\relsize{-2}]
os.path.walk(rootdir,f,arg)
\end{Verbatim}

where {\bf rootdir} is the name of the root of the desired directory tree,
{\bf f()} is a user-supplied function, and {\bf arg} will be one of the
arguments to {\bf f()}, as explained below.

At each directory {\bf d} visited in the ``walk,'' {\bf walk()} will
call {\bf f()}, and will provide {\bf f()} with the list {\bf flst} of
the names of the files in {\bf d}.  In other words, {\bf walk()} will
make this call:

\begin{Verbatim}[fontsize=\relsize{-2}]
f(arg,d,flst)
\end{Verbatim}

So, the user must write {\bf f()} to perform whatever operations she
needs for the given directory.  Remember, the user sets {\bf arg} too.
According to the Python help file for {\bf walk()}, in many applications
the user sets {\bf arg} to {\bf None} (though not in our example here).

The following example is adapted from code written by Leston
Buell\footnote{\url{http://fizzylogic.com/users/bulbul/programming/dirsize.py}},
which found the total number of bytes in a directory tree.  The
differences are:  Buell used classes to maintain the data, while I used
a list; I've added a check for linked files; and I've added code to also
calculate the number of files and directories (the latter counting the
root directory).

\begin{Verbatim}[fontsize=\relsize{-2},numbers=left]
# walkex.py; finds the total number of bytes, number of files and number
# of directories in a given directory tree, dtree (current directory if
# not specified); adapted from code by Leston Buell

# usage:
#    python walkex.py [dtree_root] 

import os, sys

def getlocaldata(sms,dr,flst):
   for f in flst:  
      # get full path name relative to where program is run; the
      # function os.path.join() adds the proper delimiter for the OS, 
      #    e.g. / for Unix, \ for Windows 
      fullf = os.path.join(dr,f)
      if os.path.islink(fullf): continue  # don't count linked files
      if os.path.isfile(fullf):
         sms[0] += os.path.getsize(fullf)
         sms[1] += 1
      else:
         sms[2] += 1

def dtstat(dtroot):
   sums = [0,0,1]  # 0 bytes, 0 files, 1 directory so far
   os.path.walk(dtroot,getlocaldata,sums)
   return sums

def main():
   try:
      root = sys.argv[1]
   except:
      root = '.'
   report = dtstat(root)
   print report

if __name__== '__main__':
   main()
\end{Verbatim}

Important feature:  When {\bf walk()} calls the user-supplied function
{\bf f()}, transmitting the list {\bf flist} of files in the
currently-visited directory, {\bf f()} may modify that list.  The key
point is that {\bf walk()} will continue to use that list to find more
directories to visit.  

For example, suppose there is a subdirectory {\bf qqq} in the
currently-visited directory.  The function {\bf f()} could delete {\bf
qqq} from {\bf flist}, with the result being that {\bf walk()} will NOT
visit the subtree having {\bf qqq} as its root.\footnote{Of course, if
it has already visited that subtree, that can't be undone.}

As another example, here is a function that descends a directory tree,
removing all empty directories:

\begin{lstlisting}[numbers=left]
# descends the directory tree rooted at rootdir, removing all empty
# directories

import os

def checklocalempty(dummyarg,dir,flst):
   for f in flst:
      fullf = os.path.join(dir,f)
      if os.path.isdir(fullf):
         subdirfls = os.listdir(fullf)
         if subdirfls == []:  os.rmdir(fullf)

def rmnulldirs(rootdir):
   fullrootdir = os.path.abspath(rootdir)
   os.path.walk(fullrootdir,checklocalempty,None)

def main():
   rmnulldirs('.')
\end{lstlisting}

\section{Cross-Platform Issues}

Like most scripting languages, Python is nominally cross-platform,
usable on Unix, Windows and Macs.  It makes a very serious attempt to
meet this goal, and succeeds reasonably well.  Let's take a closer look
at this.

\subsection{The Small Stuff}

Some aspects cross-platform compatibility are easy to deal with.  One
example of this is file path naming, e.g. {\bf /a/b/c} in Unix and {\bf
$\backslash$a$\backslash$b$\backslash$c} in Windows.  We saw above that
the library functions such as {\bf os.path.abspath()} will place slashes
or backslashes according to our underlying OS, thus enabling
platform-independent code.  In fact, the quantity {\bf os.sep} stores
the relevant character, '/' for Unix and '$\backslash$' for Windows.

Similarly, in Unix, the end-of-line marker (EOL) is a single byte,
0xa, while for Windows it is a pair of bytes, 0xd and 0xa.  The EOL
marker is stored in {\bf os.linesep}.

\subsection{How Is It Done?}

Let's take a look at the {\bf os} module, which will be in your file
{\bf /usr/lib/python2.4/os.py} or something similar.  You will see code
like

\begin{Verbatim}[fontsize=\relsize{-2}]
_names = sys.builtin_module_names
...
if 'posix' in _names:
    name = 'posix'
    linesep = '\n'
    from posix import *
    try:
        from posix import _exit
    except ImportError:
        pass
    import posixpath as path

    import posix
    __all__.extend(_get_exports_list(posix))
    del posix
\end{Verbatim}

(POSIX is the name of ``standard'' Unix.)  

\subsection{Python and So-Called ``Binary'' Files}

Keep in mind that the term {\bf binary file} is a misnomer.  After all,
ANY file is ``binary,'' whether it consists of ``text'' or not, in the
sense that it consists of bits, no matter what.  So what do people mean
when they refer to a ``binary'' file?

First, let's define the term {\bf text file} to mean a file satisifying
all of the following conditions:

\begin{itemize}

\item [(a)] Each byte in the file is in the ASCII range
00000000-01111111, i.e. 0-127.

\item [(b)] Each byte in the file is {\it intended} to be thought of as
an ASCII character.

\item [(c)] The file is {\it intended} to be broken into what we think
of (and typically display) as lines.  Here the term {\it line} is
defined technically in terms of end-of-line markers.

\end{itemize}

Any file which does not satisfy the above conditions has traditionally
been termed a {\bf binary file}.\footnote{Even this definition is
arguably too restrictive.  If we produce a non-English file which we
intend as ``text,'' it will have some non-ASCII bytes.}

The default mode of Python in reading a file is to assume it is a text
file.  Whenever an EOL marker is encountered, it will be converted to
`$\backslash$n', i.e. 0xa.  This would of course be no problem on Unix
platforms, since it would involve no change at all, but under Windows,
if the file were actually a binary file and simply by coincidence
contained some 0xd 0xa pairs, one byte from each pair would be lost, with
potentially disastrous consequences.  So, you must warn the system if it
is a binary file.  For example,

\begin{Verbatim}[fontsize=\relsize{-2}] 
f = open('yyy','rb')
\end{Verbatim}

would open the file {\bf yyy} in read-only mode, and treat the file as
binary.

% \appendix
% 
% \section{Role of the OS}
% 
% If your knowledge of operating systems is rather sketchy, you may find this
% appendix useful.
% 
% Many of the Python functions discussed above are wrappers to OS
% functions.  Let's take {\bf os.mkdir()} as an example.
% 
% What happens when you use the {\bf mkdir} command in a Unix
% shell?\footnote{The discussion here would be similar under other OSs,
% say Windows.} Keep in mind, the shell is a PROGRAM.  You didn't write
% it, but someone did (as you could if you wanted to).  Whoever wrote the
% shell put a call to the OS function {\bf mkdir()} into that program.
% When you type ``mkdir'' in a shell, the shell program reads it,
% recognizes it, and calls the {\bf mkdir()} function in the OS.  So the
% actual creation of the directory is done by the OS function {\bf
% mkdir()}, NOT by your shell command {\bf mkdir}.  
% 
% Similarly, if you call {\bf os.mkdir()} from your Python program, the
% Python interpreter will see your call, and then call {\bf mkdir()} in
% the OS on your behalf.  Python's {\bf os.mkdir()} is merely a wrapper
% for the OS call {\bf mkdir()}.
% 
% We say that {\bf mkdir()} is a {\bf system call}, i.e. a call to a
% function in the OS.  Quite a bit of what you may have thought were
% integral actions of C/C++ (and now Python) actually involve system
% calls. 
% 
% For instance, if you call {\bf printf()} or execute {\bf cout} in your
% C/C++ program, they in turn will call the {\bf write()} function in the
% OS, and {\bf scanf()} and {\bf cin} will call the OS function {\bf
% read()}.  Similar statements hold for Python's {\bf print}, {\bf
% sys.stdout.write()}, etc.