\chapter{Network Programming with Python} 
\label{chap:net}

The old ad slogan of Sun Microsystems was ``The network is the
computer.''  Though Sun has changed (now part of Oracle), the concept
has not.  The continuing computer revolution simply wouldn't exist
without networks.

\section{Overview of Networks}

The TCP/IP network protocol suite is the standard method for
intermachine communication.  Though originally integral only to the UNIX
operating system, its usage spread to all OS types, and it is the basis
of the entire Internet.  This document will briefly introduce the
subject of TCP/IP programming using the Python language.  See
\url{http://heather.cs.ucdavis.edu/~matloff/Networks/Intro/NetIntro.pdf}
for a more detailed introduction to networks and TCP/IP.  

\subsection{Networks and MAC Addresses}

A {\bf network} means a Local Area Network.  Here machines are all on a
single cable, or as is more common now, what amounts to a single cable
(e.g. multiple wires running through a switch).  The machines on the
network communicate with each other by the {\bf MAC addresses}, which
are 48-bit serial numbers burned into their network interface cards
(NICs).  If machine A wishes to send something to machine B on the same
network, A will put B's MAC address into the message packet.  B will see
the packet, recognize its own MAC address as destination, and accept the
packet.

\subsection{The Internet and IP Addresses}

An internet---note the indefinite article and the lower-case {\it
i}---is simply a connection of two or more networks.  One starts, say,
with two networks and places a computer on both of them (so it will have
two NICs).  Machines on one network can send to machines on the other by
sending to the computer in common, which is acting as a {\bf router}.
These days, many routers are not full computers, but simply boxes that
do only routing.  One of these two networks can be then connected to a
third in the same way, so we get a three-network internet, and so on.
In some cases, two networks are connected by having a machine on one
network connected to a machine on the other via a high-speed phone line,
or even a satellite connection.

{\it The} Internet---note the definite article and the capital {\it
I}---consists of millions of these networks connected in that manner.

On the Internet, every machine has an Internet Protocol (IP) address.
The original ones were 32 bits wide, and the new ones 128.  If 
machine A on one network wants to send to machine Z on an distant
network, A sends to a router, which sends to another router and so on
until the message finally reaches Z's network.  The local router there
then puts Z's MAC address in the packet, and places the packet on that
network.  Z will see it and accept the packet.

Note that when A sends the packet to a local router, the latter may not
know how to get to Z.  But it will send to another router ``in that
direction,'' and after this process repeats enough times, Z will receive
the message.  Each router has only limited information about the entire
network, but it knows enough to get the journey started.

\subsection{Ports}

The term {\it port} is overused in the computer world.  In some cases it
means something physical, such as a place into which you can plug your
USB device.  In our case here, though, ports are not physical.  Instead,
they are essentially tied to processes on a machine.

Think of our example above, where machine A sends to machine Z.  That
phrasing is not precise enough.  Instead, we should say that a process
on machine A sends to a process on machine Z.  These are identified by
ports, which are just numbers similar to file descriptors/handles.

So, when A sends to Z, it will send to a certain port at Z.  Moreover,
the packet will also state which process at A---stated in terms of a port
number at A---sent the message, so that Z knows where to send a reply.

The ports below 1024 are reserved for the famous services, for example
port 80 for HTTP.  These are called {\bf well-known ports}.\footnote{On
UNIX machines, a list of these is available in {\bf /etc/services}.  You
cannot start a server at these ports unless you are acting with root
privileges.} User programs use port numbers of 1024 or higher.  By the
way, keep in mind that a port stays in use for a few seconds after a
connection is closed; trying to start a connection at that port again
within this time period will result in an exception.

\subsection{Connectionless and Connection-Oriented Communication}

One can send one single packet at a time.  We simply state that our
message will consist of a single packet, and state the destination IP
address or port.  This is called {\bf connectionless} communication,
termed UDP under today's Internet protocol.  It's simple, but not good
for general use.  For example, a message might get lost, and the sender
would never know.  Or, if the message is long, it is subject to
corruption, which would not be caught by a connectionless setup, and
also long messages monopolize network bandwidth.\footnote{The {\bf
bandwidth} is the number of bits per second which can be sent.  It
should not be confused with {\bf latency}, which is the end-to-end
transit time for a message.}  

So, we usually use a {\bf connection-oriented} method, TCP.  What
happens here is that a message from A to Z is first broken into pieces,
which are sent separately from each other.  At Z, the TCP layer of the
network protocol stack in the OS will collect all the pieces, check them
for errors, put them in the right order, etc. and deliver them to the
proper process at Z.

We say that there is a {\bf connection} between A and Z.  Again, this is
not physical.  It merely is an agreement between the OSs at A and Z that
they will exchange data between these processes/ports at A and Z in the
orderly manner described above.

{\bf One point which is crucial to keep in mind is that under TCP,
everything from machine A to Z, from the time the connection is opened
to the time it is closed is considered one gigantic message.}  If the
process at A, for instance, executes three network writes of 100 bytes
each, it is considered one 300-byte message.  And though that message
may be split into pieces along the way to Z, the piece size will almost
certainly not be 100 bytes, nor will the number of pieces likely be
three.  Of course, the same is true for the material sent from Z to A
during this time.

This makes writing the program on the Z side more complicated.  Say for
instance A wishes to send four lines of text, and suppose the program at
B knows it will be sent four lines of text.  Under UDP, it would be
natural to have the program at A send the data as four network writes,
and the program at B would do four network reads.

But under TCP, no matter how many or few network writes the program at A 
does, the program at B will not know how many network reads to do.  So,
it must keep reading in a {\bf while} loop until it receives a messages
saying that A has nothing more to send and has closed the connection.
This makes the program at Z harder to write.  For example, that program
must break the data into lines on its own.  (I'm talking about {\it user}
programs here; in other words, TCP means more trouble for {\it you}.)

\subsection{Clients and Servers}

Connections are not completely symmetric.\footnote{I'm speaking mainly
of TCP here, but it mostly applies to UDP too.} Instead, a connection
consists of a {\bf client} and a {\bf server}.  The latter sits around
waiting for requests for connections, while the former makes such a
request.

When you surf the Web, say to {\bf http://www.google.com}, your Web
browser is a client.  The program you contact at Google is a server.
When a server is run, it sets up business at a certain port, say 80 in
the Web case.  It then waits for clients to contact it.  When a client
does so, the server will create a new socket, specifically for
communication with that client, and then resume watching the original
socket for new requests.

\section{Our Example Client/Server Pair}

As our main illustration of client/server programming in Python, we have
modified a simple example in the Library Reference section of the Python
documentation page, \url{http://www.python.org/doc/current/lib}.  Here
is the server, {\bf tms.py}:

\begin{Verbatim}[fontsize=\relsize{-2},numbers=left]
# simple illustration client/server pair; client program sends a string
# to server, which echoes it back to the client (in multiple copies),
# and the latter prints to the screen

# this is the server

import socket
import sys

# create a socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# associate the socket with a port
host = ''  # can leave this blank on the server side
port = int(sys.argv[1])
s.bind((host, port))

# accept "call" from client
s.listen(1)
conn, addr = s.accept()
print 'client is at', addr

# read string from client (assumed here to be so short that one call to
# recv() is enough), and make multiple copies (to show the need for the
# "while" loop on the client side)

data = conn.recv(1000000)
data = 10000 * data  # concatenate data with itself 999 times

# wait for the go-ahead signal from the keyboard (to demonstrate that 
# recv() at the client will block until server sends)
z = raw_input()

# now send 
conn.send(data)

# close the connection
conn.close()
\end{Verbatim}

And here is the client, {\bf tmc.py}:

\begin{Verbatim}[fontsize=\relsize{-2},numbers=left]
# simple illustration client/server pair; client program sends a string
# to server, which echoes it back to the client (in multiple copies),
# and the latter prints to the screen

# this is the client

import socket
import sys

# create a socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# connect to server
host = sys.argv[1]  # server address
port = int(sys.argv[2])  # server port
s.connect((host, port))

s.send(sys.argv[3])  # send test string

# read echo
i = 0
while(1):
   data = s.recv(1000000)  # read up to 1000000 bytes 
   i += 1
   if (i < 5):  # look only at the first part of the message
      print data
   if not data:  # if end of data, leave loop
      break
   print 'received', len(data), 'bytes'

# close the connection
s.close()
\end{Verbatim}

This client/server pair doesn't do much.  The client sends a test string
to the server, and the server sends back multiple copies of the string.
The client then prints the earlier part of that echoed material to the
user's screen, to demonstrate that the echoing is working, and also
prints the amount of data received on each read, to demonstrate the
``chunky'' nature of TCP discussed earlier. 

You should run this client/server pair before reading
further.\footnote{The source file from which this document is created,
{\bf PyNet.tex}, should be available wherever you downloaded the PDF
file.  You can get the client and server programs from the source file,
rather than having to type them up yourself.}  Start up the server on
one machine, by typing

\begin{verbatim}
python tms.py 2000
\end{verbatim}

and then start the client at another machine, by typing

\begin{verbatim}
python tmc.py server_machine_name 2000 abc
\end{verbatim}

(Make sure to start the server before you start the client!)

The two main points to note when you run the programs are that (a) the
client will block until you provide some keyboard input at the server
machine, and (b) the client will receive data from the server in rather
random-sized chunks.

\subsection{Analysis of the Server Program}

Let's look at the details of the server.  

{\bf Line 7:}  We import the {\bf socket} class from Python's
library; this contains all the communication methods we need.  

{\bf Line 11:}  We create a socket.  This is very much analogous to a
file handle or file descriptor in applications involving files.
Internally it is a pointer to information about our connection (not yet
made) to an application on another machine.  Again, at this point, it is
merely a placeholder.  We have not undertaken any network actions yet.
% But our calls {\bf bind()} etc. below will result in more and more
% information being added to the place pointed to by our socket. 

The two arguments state that we wish to the socket to be an Internet
socket ({\bf socket.AF\_INET}), and that it will use TCP ({\bf
socket.SOCK\_STREAM}), rather than UDP ({\bf socket.SOCK\_DGRAM}).  Note
that the constants used in the arguments are attributes of the module
{\bf socket}, so they are preceded by `socket.'; in C/C++, the analog is
the {\bf \#include} file.

{\bf Line 16:}  We invoke the {\bf socket} class' {\bf bind()} method.
Say for example we specify port 2000 on the command line when we run the
server (obtained on Line 15).

When we call {\bf bind()}, the operating system will first check to see
whether port 2000 is already in use by some other process.\footnote{This
could be another invocation of our server program, or a different
program entirely.  You could check this ``by hand,'' by running the UNIX
{\bf netstat} command (Windows has something similar), but it would be
better to have your program do it, using a Python {\bf try}/{\bf except}
construct.}  If so, an exception will be raised, but otherwise the OS
will reserve port 2000 for the server.  What that means is that from now
on, whenever TCP data reaches this machine and specifies port 2000, that
data will be copied to our server program.  Note that {\bf bind()} takes
a single argument consisting of a two-element tuple, rather than two
scalar arguments.

{\bf Line 19:}  The {\bf listen()} method tells the OS that if any
messages come in from the Internet specifying port 2000, then they
should be considered to be requesting a connection to this socket.

The method's argument tells the OS how many connection requests from
remote clients to allow to be pending at any give time for port 2000.
The argument 1 here tells the OS to allow only 1 pending connection
request at a time.  

We only care about one connection in this application, so we set the
argument to 1.  If we had set it to, say 5 (which is common), the OS
would allow one active connection for this port, and four other pending
connections for it.  If a fifth pending request were to come it, it
would be rejected, with a ``connection refused'' error. 

That is about all {\bf listen()} really does.   

We term this socket a {\bf listening socket}.  That means its sole
purpose is to accept connections with clients; it is usually not used
for the actual transfer of data back and forth between clients and the
server.\footnote{It could be used for that purpose, if our server only
handles one client at a time.}

{\bf Line 20:}  The {\bf accept()} method tells the OS to wait for a
connection request.  It will block until a request comes in from a
client at a remote machine.\footnote{Which, technically, could be the
same machine.} That will occur when the client executes a {\bf
connect()} call (Line 16 of {\bf tmc.py}).  In that call, the OS at the
client machine sends a connection request to the server machine,
informing the latter as to (a) the Internet address of the client
machine and (b) the {\bf ephemeral} port of the client.\footnote{The
term here alludes to the temporary nature of this port, compared to the
server port, which will extend over the span of all the clients the
server deals with.}

At that point, the connection has been established.  The OS on the
server machine sets up a new socket, termed a {\bf connected socket},
which will be used in the server's communication with the remote client.

You might wonder why there are separate listening and connected sockets.
Typically a server will simultaneously be connected to many clients.  So
it needs a separate socket for communication with each client.  (It then
must either set up a separate thread for each one, or use nonblocking
I/O.  More on the latter below.)

All this releases {\bf accept()} from its blocking status, and it
returns a two-element tuple.  The first element of that tuple, assigned
here to {\bf conn}, is the connected socket.  Again, this is what will
be used to communicate with the client (e.g. on Line 35).

The second item returned by {\bf accept()} tells us who the client is,
i.e. the Internet address of the client, in case we need to know
that.\footnote{When I say ``we,'' I mean ``we, the authors of this
server program.''  That information may be optional for us, though
obviously vital to the OS on the machine where the server is running.
The OS also needs to know the client's ephemeral port, while ``we''
would almost never have a need for that.}

{\bf Line 27:}  The {\bf recv()} method reads data from the given
socket.  The argument states the maximum number of bytes we are willing
to receive.  This depends on how much memory we are willing to
have our server use.  It is traditionally set at 1024. 

It is absolutely crucial, though, to keep in mind how TCP works in this
regard.  To review, consider a connection set up between a client X and
server Y.  The entirety of data that X sends to Y is considered one
giant message.  If for example X sends text data in the form of 27 lines
totalling 619 characters, TCP makes no distinction between one line and
another; TCP simply considers it to be one 619-byte message.

Yet, that 619-byte message might not arrive all at once.  It might, for
instance, come into two pieces, one of 402 bytes and the other of 217
bytes.  And that 402-byte piece may not consist of an integer
number of lines.  It may, and probably would, end somewhere in the
middle of a line.  For that reason, we seldom see a one-time call to
{\bf recv()} in real production code, as we see here on Line 27.
Instead, the call is typically part of a loop, as can be seen starting
on Line 22 of the client, {\bf tmc.py}.  In other words, here on Line 27
of the server, we have been rather sloppy, going on the assumption that
the data from the client will be so short that it will arrive in just
one piece.  In a production program, we would use a loop.

{\bf Line 28:}  In order to show the need for such a loop in general, I have
modified the original example by making the data really long.  Recall
that in Python, ``multiplying'' a string means duplicating it.  For
example:

\begin{verbatim}
>>> 3*'abc'
'abcabcabc'
\end{verbatim}

Again, I put this in deliberately, so as to necessitate using a loop in
the client, as we will see below.

{\bf Line 32:}  This too is inserted for the purpose of illustrating a
principle later in the client.  It takes some keyboard input at the
server machine.  The input is not actually used; it is merely a stalling
mechanism.

{\bf Line 35:}  The server finally sends its data to the client.

{\bf Line 38:}  The server closes the connection.  At this point, the
sending of the giant message to the client is
complete.\footnote{However, that doesn't necessarily mean that the
message has arrived at the client yet, nor even that the message has
even left the server's machine yet.  See below.}  The closing of the
connection will be sensed by the client, as discussed below.

\subsection{Analysis of the Client Program}

Now, what about the client code?

{\bf Line 16:}  The client makes the connection with the server.  Note
that both the server's Internet machine address and the server's port
number are needed.  As soon as this line is executed, Line 20 on the
server side, which had been waiting, will finally execute.

Again, the connection itself is not physical.  It merely is an agreement
made between the server and client to exchange data, in agreed-upon
chunk sizes, etc.

{\bf Line 18:}  The client sends its data to the server.

{\bf Lines 22ff:}  The client reads the message from the server.  As
explained earlier, this is done in a loop, because the message is likely
to come in chunks.  Again, even though Line 35 of the server gave the
data to its OS in one piece, the OS may not send it out to the network
in one piece, and thus the client must loop, repeatedly calling {\bf
recv()}.  

That raises the question of how the client will know that it has
received the entire message sent by the server.  The answer is that {\bf
recv()} will return an empty string when that occurs.  And in turn, that
will occur when the server executes {\bf close()} on Line
38.\footnote{This would also occur if {\bf conn} were to be
garbage-collected when its scope ended, including the situation in which
the server exits altogether.}

Note:  

\begin{itemize}

\item  In Line 23, the client program is basically saying to the OS,
``Give me whatever characters you've received so far.''  If the OS
hasn't received any characters yet (and if the connection has not been
closed), {\bf recv()} will block until something does come
in.\footnote{This will not be the case if the socket is {\bf
nonblocking}.  More on this in Section \ref{nb}.} 

\item When the server finally does close the connection {\bf recv()}
will return an empty string. 

\end{itemize}

\section{Role of the OS}

\subsection{Basic Operation}

As is the case with file functions, e.g. {\bf os.open()}, the functions
{\bf socket.socket()}, {\bf socket.bind()}, etc. are all wrappers to OS
system calls.

The Python {\bf socket.send()} calls the OS {\bf send()}.  The latter
copies the data (which is an argument to the function) to the OS'
buffer.  Again, assuming we are using TCP, the OS will break the message
into pieces before putting the data in the buffer.  Characters from the
latter are at various times picked up by the Ethernet card's device
driver and sent out onto the network.  

When a call to {\bf send()} returns, that simply means that at least
part of the given data has been copied from the application program to
the OS' buffer.  It does not mean that ALL of the data has been copied
to the buffer, let alone saying that the characters have actually gotten
onto the network yet, let alone saying they have reached the receiving
end's OS, let alone saying they have reached the receiving end's
application program.  The OS will tell us how many bytes it accepted
from us to send out onto the network, via the return value from the call
to {\bf send()}.  (So, technically even {\bf send()} should be in a
loop, which iterates until all of our bytes have been accepted.  See
below.)

The OS at the receiving end will receive the data, check for errors and
ask the sending side to retransmit an erroneous chunk, piece the data
back to together and place it in the OS' buffer.  Each call to {\bf
recv()} by the application program on the receiving end will pick up
whatever characters are currently in the buffer (up to the number
specified in the argument to {\bf recv()}).

\subsection{How the OS Distinguishes Between Multiple Connections}

When the server accepts a connection from a client, the connected socket
will be given the same port as the listening socket.  So, we'll have two
different sockets, both using the same port.  If the server has
connections open with several clients, and the associated connected
sockets all use the same port, how does the OS at the server machine
decide which connected socket to give incoming data to for that port?

The answer lies in the fact that a connection is defined by five
numbers:  The server port; the client (ephemeral) port; the server IP
address; the client IP address; and the protocol (TCP or UDP).
Different clients will usually have different Internet addresses, so
that is a distinguishing aspect.  But even more importantly, two clients
could be on the same machine, and thus have the same Internet address,
yet still be distinguished from each other by the OS at the server
machine, because the two clients would have different ephemeral
addresses.  So it all works out.

\section{The {\tt sendall()} Function}

We emphasized earlier why a call to {\bf recv()} should be put in a
loop.  One might also ask whether {\bf send()} should be put in a loop
too.

Unless the socket is nonblocking, {\bf send()} will block until the OS
on our machine has enough buffer space to accept at least some of the
data given to it by the application program via {\bf send()}.  When it
does so, the OS will tell us how many bytes it accepted, via the return
value from the call to {\bf send()}.  The question is, is it possible
that this will not be all of the bytes we wanted to send?  If so, we
need to put {\bf send()} in a loop, e.g.  something like this, where we
send a string {\bf w} via a socket {\bf s}:

\begin{Verbatim}[fontsize=\relsize{-2}]
while(len(w) > 0):
   ns = s.send(w)  # ns will be the number of bytes sent
   w = w[ns:]  # still need to send the rest
\end{Verbatim}

The best reference on TCP/IP programming ({\it UNIX Network
Programming}, by Richard Stevens, pub.  Prentice-Hall, vol. 1, 2nd ed.,
p.77) says that this problem is ``normally'' seen only if the socket is
nonblocking.  However, that was for UNIX, and in any case, the best he
seemed to be able to say was ``normally.''  To be fully safe, one should
put one's call to {\bf send()} inside a loop, as shown above.

But as is often the case, Python recognizes that this is such a common
operation that it should be automated.  Thus Python provides the {\bf
sendall()} function.  This function will not return until the entire
string has been sent (in the sense stated above, i.e. completely copied
to the OS' buffer).

The function {\bf sendall()} should be used only with blocking sockets.

\section{Sending Lines of Text} 

\subsection{Remember, It's Just One Big Byte Stream, Not ``Lines''}

As discussed earlier, TCP regards all the data sent by a client or
server as one giant message.  If the data consists of lines of text, TCP
will not pay attention to the demarcations between lines.  This means
that if your application is text/line-oriented, you must handle such
demarcation yourself.  If for example the client sends lines of text to
the server, your server code must look for newline characters and
separate lines on its own.  Note too that in one call to {\bf recv()} we
might receive, say, all of one line and part of the next line, in which
case we must keep the latter for piecing together with the bytes we get
from our next call to {\bf recv()}.  This becomes a nuisance for the
programmer.

In our example above, the client and server each execute {\bf send()}
only once, but in many applications they will alternate.  The client
will send something to the server, then the server will send something
to the client, then the client will send something to the server, and so
on.  In such a situation, it will \underline{still} be the case that the
totality of all bytes sent by the client will be considered one single
message by the TCP/IP system, and the same will be true for the server.  

\subsection{The Wonderful {\tt makefile()} Function}

As mentioned, if you are transferring text data between the client and
server (in either direction), you've got to piece together each line on
your own, a real pain.  Not only might you get only part of a line
during a receive, you might get part of the {\it next} line, which you
would have to save for later use with reading the next line.
\footnote{One way around this problem would be to read one byte at a
time, i.e. call {\bf recv()} with argument 1.  But this would be very
inefficient, as system calls have heavy overhead.}  But Python allows
you to avoid this work, by using the method {\bf socket.makefile()}.

Python has the notion of a {\it file-like object}.  This is a byte
stream that you can treat as a ``file,'' thinking of it as consisting of
``lines.''  For example, we can invoke {\bf readlines()}, a file
function, on the standard input:

\begin{Verbatim}[fontsize=\relsize{-2}]
>>> import sys
>>> w = sys.stdin.readlines()  # ctrl-d to end input
moon, sun
and
stars
>>> w
['moon, sun\n', 'and\n', 'stars\n']
\end{Verbatim}

Well, {\bf socket.makefile()} allows you to do this with sockets, as
seen in the following example.  

Here we have a server which will allow anyone on the Internet to find
out which processes are running on the host machine---even if they don't
have an account on that machine.\footnote{Some people have trouble
believing this.  How could you access such information without even
having an account on that machine?  Well, the server is willing to give
it to you.  Imagine what terrible security problems we'd have if the
server allowed one to run any command from the client, rather than just
{\bf w} and {\bf ps}.}  See the comments at the beginning of the
programs for usage.

Here is the client:

\begin{Verbatim}[fontsize=\relsize{-2},numbers=left]
# wps.py

# client for the server for remote versions of the w and ps commands

# user can check load on machine without logging in (or even without
# having an account on the remote machine)

# usage:  

#   python wps.py remotehostname port_num {w,ps} 

# e.g. python wps.py nimbus.org 8888 w would cause the server at
# nimbus.org on port 8888 to run the UNIX w command there, and send the
# output of the command back to the client here

import socket,sys

def main():

   s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
   host = sys.argv[1]  
   port = int(sys.argv[2])  
   s.connect((host,port))

   # send w or ps command to server
   s.send(sys.argv[3])

   # create "file-like object" flo
   flo = s.makefile('r',0)  # read-only, unbuffered
   # now can call readlines() on flo, and also use the fact that
   # that stdout is a file-like object too
   sys.stdout.writelines(flo.readlines())

if __name__ == '__main__':
   main()
\end{Verbatim}

And here is the server:

\begin{Verbatim}[fontsize=\relsize{-2},numbers=left]
# svr.py

# server for remote versions of the w and ps commands

# user can check load on machine without logging in (or even without
# having an account on the remote machine)

# usage:

#   python svr.py port_num 

import socket,sys,os

def main():

   ls = socket.socket(socket.AF_INET,socket.SOCK_STREAM);
   port = int(sys.argv[1])
   ls.bind(('', port))

   while (1): 
      ls.listen(1)
      (conn, addr) = ls.accept()
      print 'client is at', addr
      # get w or ps command from client
      rc = conn.recv(2)
      # run the command in a Unix-style pipe
      ppn = os.popen(rc)  # do ppn.close() to close
      # ppn is a "file-like object," so can apply readlines()
      rl = ppn.readlines()
      # create a file-like object from the connection socket
      flo = conn.makefile('w',0)  # write-only, unbuffered
      # write the lines to the network
      flo.writelines(rl[:-1])
      # clean up 
      # must close both the socket and the wrapper
      flo.close()
      conn.close()

if __name__ == '__main__':
   main()
\end{Verbatim}

Note that there is no explicit call to {\bf recv()} in the client code.
The call to {\bf readlines()} basically triggers such a call.  And the
reason we're even allowed to call {\bf readlines()} is that we called
{\bf makefile()} and created {\bf flo}.

\subsection{Getting the Tail End of the Data}

A common bug in network programming is that most of a data transfer
works fine, but the receiving program hangs at the end, waiting to
receive the very last portion of the data.  The cause of this is that,
for efficiency reasons, the TCP layer at the sending end generally waits
until it accumulates a ``large enough'' chunk of data before it sending
out.  As long as the sending side has not closed the connection, the TCP
layer there assumes that the program may be sending more data, and TCP
will wait for it.

The simplest way to deal with this is for the sending side to close the
connection.  Its TCP layer then knows that it will be given no more data
by the sending program, and thus TCP needs to send out whatever it has.
This means that both client and server programs must do a lot of opening
and closing of socket connections, which is inefficient.\footnote{Yet
such inefficiency is common, and is used for example in the HTTP (i.e.
Web access) protocol.  Each time you click the mouse on a given Web
page, for instance, there is a new connection made.  It is this lack of
{\bf state} in the protocol that necessitates the use of {\bf cookies}.
Since each Web action involves a separate connection, there is no
``memory'' between the actions.  This would mean, for example, that if
the Web site were password-protected, the server would have to ask you
for your password at every single action, quite a nuisance.  The
workaround is to have your Web browser write a file to your local disk,
recording that you have already passed the password test.}
 
Also, if you are using {\bf makefile()}, it would be best to not use
{\bf readlines()} directly, as this may cause the receiving end to wait
until the sender closes the connection, so that the ``file'' is
complete.  It may be safer to do something like this:

\begin{Verbatim}[fontsize=\relsize{-2}]
# flo is an already-created file-like object from calling makefile()
for line in flo:
   ...
\end{Verbatim}

This way we are using {\bf flo} as an iterator, which will not require
all lines to be received before the loop is started.

An alternative is to use UDP instead of TCP.  With UDP, if you send an
n-byte message, it will all be send immediately, in one piece, and
received as such.  However, you then lose the reliability of
TCP.\footnote{TCP does error checking, including checking for lost
packets.  If the client and server are multiple hops apart in the
Internet, it's possible that some packets will be lost when an
intermediate router has buffer overflow.  If TCP at the receiving end
doesn't receive a packet by a certain {\bf timeout} period, it will ask
TCP at the sending end to retransmit the packet.  Of course, we could do
all this ourselves in UDP by adding complexity to our code, but that
would defeat the purpose.}

A related problem with {\bf popen()} is discussed at
\url{http://www.popekim.com/2008/12/never-use-pipe-with-python-popen.html}.

\section{Dealing with Asynchronous Inputs}

In many applications a machine, typically a server, will be in a
position in which input could come from several network sources, without
knowing which one will come next.  One way of dealing with that is to
take a threaded approach, with the server having a separate thread for
each possible client.  Another way is to use nonblocking sockets.

\subsection{Nonblocking Sockets}
\label{nb}

Note our statement that {\bf recv()} blocks until either there is data
available to be read or the sender has closed the connection holds only
if the socket is in blocking mode.  That mode is the default, but we can
change a socket to nonblocking mode by calling {\bf setblocking()} with
argument 0.\footnote{As usual, this is done in a much cleaner, easier
manner than in C/C++.  In Python, one simple function call does it.}  

One calls {\bf recv()} in nonblocking mode as part of a {\bf try}/{\bf
except} pair.  If data is available to be read from that socket,
{\bf recv()} works as usual, but if no data is available, an exception
is raised.  While one normally thinks of exceptions as cases in which a
drastic execution error has occurred, in this setting it simply means
that no data is yet available on this socket, and we can go to try
another socket or do something else.

Why would we want to do this?  Consider a server program which is
connected to multiple clients simultaneously.  Data will come in from
the various clients at unpredictable times.  The problem is that if we
were to simply read from each one in order, it could easily happen that
{\bf read()} would block while reading from one client while data from
another client is ready for reading.  Imagine that you are the latter
client, while the former client is out taking a long lunch!  You'd be
unable to use the server all that time!

One way to handle this is to use threads, setting up one thread for each
client.  Indeed, I have an example of this in Chapter \ref{chap:pythr}.
But threads programming can be tricky, so one may turn to the
alternative, which is to make the client sockets nonblocking.

The example below does nothing useful, but is a simple illustration of
the principles.  Each client keeps sending letters to the server; the
server concatenates all the letters it receives, and sends the
concatenated string back to a client whenever the client sends a
character.

In a sample run of these programs, I started the server, then one client
in one window, then another client is another window.  (For convenience,
I was doing all of this on the same machine, but a better illustration
would be to use three different machines.) In the first window, I typed
'a', then 'b', then 'c'.  Then I moved to the other window, and typed
'u', 'u', 'v' and 'v'.  I then went back to the first window and typed
'd', etc.  I ended each client session by simply hitting Enter instead
of typing a letter.

Here is what happened at the terminal for the first client:

\begin{Verbatim}[fontsize=\relsize{-2},numbers=left]
% python strclnt.py localhost 2000
enter a letter:a
a
enter a letter:b
ab
enter a letter:c
abc
enter a letter:d
abcuuvvd
enter a letter:e
abcuuvvdwwe
enter a letter:
\end{Verbatim}

Here is what happened at the terminal for the second client:

\begin{Verbatim}[fontsize=\relsize{-2},numbers=left]
% python strclnt.py localhost 2000
enter a letter:u
abcu
enter a letter:u
abcuu
enter a letter:v
abcuuv
enter a letter:v
abcuuvv
enter a letter:w
abcuuvvdw
enter a letter:w
abcuuvvdww
enter a letter:
\end{Verbatim}

Here is what happened at the terminal for the server:

\begin{Verbatim}[fontsize=\relsize{-2},numbers=left]
% python strsvr.py 2000
the final value of v is abcuuvvdwwe
\end{Verbatim}

Note that I first typed at the first client, but after typing 'c', I
switched to the second client.  After hitting the second 'v', I switched
back to the first, etc. 

Now, let's see the code.  First, the client:

\begin{Verbatim}[fontsize=\relsize{-2},numbers=left]
# strclnt.py:  simple illustration of nonblocking sockets

# two clients connect to server; each client repeatedly sends a letter k,
# which the server appends to a global string v and reports it to the
# client; k = '' means the client is dropping out; when all clients are
# gone, server prints the final string v

# this is the client; usage is
#    python clnt.py server_address server_port_number

import socket, sys 

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = sys.argv[1]  # server address
port = int(sys.argv[2])  # server port
s.connect((host, port))

while 1:
   # get letter
   k = raw_input('enter a letter:')
   s.send(k)  # send k to server
   # if stop signal, then leave loop
   if k == '': break
   v = s.recv(1024)  # receive v from server (up to 1024 bytes, assumed
                     # to come in one chunk)
   print v  

s.close() # close socket
\end{Verbatim}

And here is the server:

\begin{Verbatim}[fontsize=\relsize{-2},numbers=left]
# strsvr.py:  simple illustration of nonblocking sockets

# multiple clients connect to server; each client repeatedly sends a
# letter k, which the server adds to a global string v and echos back
# to the client; k = '' means the client is dropping out; when all
# clients are gone, server prints final value of v

# this is the server; usage is
#    python strsvr.py server_port_number

import socket, sys  

# set up listening socket
lstn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

port = int(sys.argv[1]) 
# bind lstn socket to this port 
lstn.bind(('', port))
lstn.listen(5)

# initialize concatenate string, v
v = ''

# initialize client socket list
cs = []

# in this example, a fixed number of clients, nc
nc = 2

# accept connections from the clients
for i in range(nc):
   (clnt,ap) = lstn.accept()
   # set this new socket to nonblocking mode
   clnt.setblocking(0)
   cs.append(clnt)

# now loop, always accepting input from whoever is ready, if any, until
# no clients are left
while len(cs) > 0:
   # get next client, with effect of a circular queue
   clnt = cs.pop(0)
   cs.append(clnt)
   # something ready to read from clnt? clnt closed connection?
   try:  
      k = clnt.recv(1)  # try to receive one byte; if none is ready
                        # yet, that is the "exception"
      if k == '': 
         clnt.close()
         cs.remove(clnt)
      v += k  # update the concatenated list
      clnt.send(v)
   except:  pass

lstn.close()
print 'the final value of v is', v
\end{Verbatim}

Here we only set the client sockets in nonblocking mode, not the
listening socket.  However, if we wished to allow the number of clients
to be unknown at the time execution starts, rather than a fixed number
known ahead of time as in our example above, we would be forced to
either make the listening socket nonblocking, or use threads.

Note once again that the actual setting of blocking/nonblocking mode is
done by the OS.  Python's {\bf setblocking()} function merely makes
system calls to make this happen.  It should be said, though, that this
Python function is far simpler to use than what must be done in C/C++ to
set the mode.

\subsection{Advanced Methods of Polling}

In our example of nonblocking sockets above, we had to ``manually''
check whether a socket was ready to read.  Today most OSs can automate
that process for you.  The ``traditional'' way to do this was via a UNIX
system call named {\bf select()}, which later was adopted by Windows as
well.  The more modern way for UNIX is another call, {\bf poll()} (not
yet available on Windows).  Python has interfaces to both of these
system calls, in the {\bf select} module.  Python also has modules {\bf
asyncore} and {\bf asynchat} for similar purposes.  I will not give the
details here.

\section{Troubleshooting}

Network program debugging can be tricky.  Here are some tips:

\begin{itemize}

\item As always, be sure to use a debugging tool!

\item The {\bf socket.getsockname()} can be used to determine the IP
address and port number of a given socket, and {\bf socket.getpeername()}
will give you the same information for this socket's {\bf peer}, i.e.
the machine on the other end of the TCP connection.

\item The following Unix commands list current connections:

\begin{Verbatim}[fontsize=\relsize{-2}]
netstat -A inet
lsof -i
\end{Verbatim}

\end{itemize}

A program which has an open socket {\bf s} can determine its own IP
address and port number by the call {\bf s.getsockname()}.

\section{Other Libraries}

Python has libraries for FTP, HTML processing and so on.  In addition, a
higher-level library which is quite popular is Twisted.

\section{Web Operations}

There is a wealth of library code available for Web operations.  Here we
look at only a short introductory program covering a couple of the many
aspects of this field:

\begin{Verbatim}[fontsize=\relsize{-2},numbers=left]
# getlinks.py:  illustrates HTML parsing in Python

import urllib  # library for accessing Web sites
import HTMLParser  # library for parsing HTML

# basic idea:  Python provides the HTMLParser class within the
# HTMLParser module; the idea is to subclass it, overriding some of the
# member functions, in this case handle_starttag()
class GetLinks(HTMLParser.HTMLParser):
   # arguments to handle_starttag(), supplied by the class during its
   # parsing actions:
   # tag:  an HTML tag, e.g. 'a' arising from '<a href>'
   # attrs:  a two-tuples, again supplied by the class during
   # parsing, that give the value for the given HTML name;
   # e.g. when the class encounters 
   # '<a href="http://heather.cs.ucdavis.edu/search.html">'
   # tag will be 'a', name will be 'href' and value will be
   # 'http://heather.cs.ucdavis.edu/search.html'
   def handle_starttag(self,tag,attrs):
      if tag == 'a':
         for name,value in attrs:
            if name == 'href': print value  # print URL link

gl = GetLinks()  # construct a class instance
url = 'http://www.ucdavis.edu/index.html'  # target URL
# open network connection; returns a file-like object, so can be read
urlconn = urllib.urlopen(url)  
# read, and put the downloaded HTML code into urlcontents
urlcontents = urlconn.read()  
# input the downloaded material into HTMLParser's member function feed
# for parsing
gl.feed(urlcontents)
\end{Verbatim}

The program reads the raw HTML code from a Web page, and then extracts
the URL links.  The comments explain the details.