\chapter{Introduction to Parallel Processing} \label{chap:intro} Parallel machines provide a wonderful opportunity for applications with large computational requirements. Effective use of these machines, though, requires a keen understanding of how they work. This chapter provides an overview. \section{Overview: Why Use Parallel Systems?} \subsection{Execution Speed} There is an ever-increasing appetite among some types of computer users for faster and faster machines. This was epitomized in a statement by Steve Jobs, founder/CEO of Apple and Pixar. He noted that when he was at Apple in the 1980s, he was always worried that some other company would come out with a faster machine than his. But now at Pixar, whose graphics work requires extremely fast computers, he is always \underbar{hoping} someone produces faster machines, so that he can use them! A major source of speedup is the parallelizing of operations. Parallel operations can be either within-processor, such as with pipelining or having several ALUs within a processor, or between-processor, in which many processor work on different parts of a problem in parallel. Our focus here is on between-processor operations. For example, the Registrar's Office at UC Davis uses shared-memory multiprocessors for processing its on-line registration work. Online registration involves an enormous amount of database computation. In order to handle this computation reasonably quickly, the program partitions the work to be done, assigning different portions of the database to different processors. The database field has contributed greatly to the commercial success of large shared-memory machines. As the Pixar example shows, highly computation-intensive applications like computer graphics also have a need for these fast parallel computers. No one wants to wait hours just to generate a single image, and the use of parallel processing machines can speed things up considerably. For example, consider \textbf{ray tracing} operations. Here our code follows the path of a ray of light in a scene, accounting for reflection and absorbtion of the light by various objects. Suppose the image is to consist of 1,000 rows of pixels, with 1,000 pixels per row. In order to attack this problem in a parallel processing manner with, say, 25 processors, we could divide the image into 25 squares of size 200x200, and have each processor do the computations for its square. Note, though, that it may be much more challenging than this implies. First of all, the computation will need some communication between the processors, which hinders performance if it is not done carefully. Second, if one really wants good speedup, one may need to take into account the fact that some squares require more computation work than others. More on this below. In this setting you need the program to run as fast as possible. Thus, in order to write good parallel processing software, you must have a good knowledge of the underlying hardware. You must find clever tricks for \textbf{load balancing,} i.e. keeping all the processors busy as much as possible. In the graphics ray-tracing application, for instance, suppose a ray is coming from the ``northeast'' section of the image, and is reflected by a solid object. Then the ray won't reach some of the ``southwest'' portions of the image, which then means that the processors assigned to those portions will not have any work to do which is associated with this ray. What we need to do is then try to give these processors some other work to do; the more they are idle, the slower our system will be. \subsection{Memory} Yes, execution speed is the reason that comes to most people's minds when the subject of parallel processing comes up. But in many applications, an equally important consideration is memory capacity. Parallel processing application often tend to use huge amounts of memory, and in many cases the amount of memory needed is more than can fit on one machine. If we have many machines working together, especially in the message-passing settings described below, we can accommodate the large memory needs. \section{Parallel Processing Hardware} This is not a hardware course, but since the goal of using parallel hardware is speed, the efficiency of our code is a major issue. That in turn means that we need a good understanding of the underlying hardware that we are programming. In this section, we give an overview of parallel hardware. \subsection{Shared-Memory Systems} \subsubsection{Basic Architecture} Here many CPUs share the same physical memory. This kind of architecture is sometimes called MIMD, standing for Multiple Instruction (different CPUs are working independently, and thus typically are executing different instructions at any given instant), Multiple Data (different CPUs are generally accessing different memory locations at any given time). Until recently, shared-memory systems cost hundreds of thousands of dollars and were affordable only by large companies, such as in the insurance and banking industries. The high-end machines are indeed still quite expensive, but now {\bf dual-core} machines, in which two CPUs share a common memory, are commonplace in the home. \subsubsection{Example: SMP Systems} A Symmetric Multiprocessor (SMP) system has the following structure: \includegraphics{Images/UMABus.jpg} Here and below: \begin{itemize} \item The Ps are processors, e.g. off-the-shelf chips such as Pentiums. \item The Ms are \textbf{memory modules}. These are physically separate objects, e.g. separate boards of memory chips. It is typical that there will be the same number of memory modules as processors. In the shared-memory case, the memory modules collectively form the entire shared address space, but with the addresses being assigned to the memory modules in one of two ways: \begin{itemize} \item (a) High-order interleaving. Here consecutive addresses are in the \underbar{same} M (except at boundaries). For example, suppose for simplicity that our memory consists of addresses 0 through 1023, and that there are four Ms. Then M0 would contain addresses 0-255, M1 would have 256-511, M2 would have 512-767, and M3 would have 768-1023. We need 10 bits for addresses (since $1024 = 2^{10}$). The two most-significant bits would be used to select the module number (since $4 = 2^2$); hence the term {\it high-order} in the name of this design. The remaining eight bits are used to select the word within a module. \item (b) Low-order interleaving. Here consecutive addresses are in consecutive memory modules (except when we get to the right end). In the example above, if we used low-order interleaving, then address 0 would be in M0, 1 would be in M1, 2 would be in M2, 3 would be in M3, 4 would be back in M0, 5 in M1, and so on. Here the two least-significant bits are used to determine the module number. \end{itemize} \item To make sure only one processor uses the bus at a time, standard bus arbitration signals and/or arbitration devices are used. \item There may also be \textbf{coherent caches}, which we will discuss later. \end{itemize} \subsection{Message-Passing Systems} \subsubsection{Basic Architecture} Here we have a number of independent CPUs, each with its own independent memory. The various processors communicate with each other via networks of some kind. \subsubsection{Example: Networks of Workstations (NOWs)} Large shared-memory multiprocessor systems are still very expensive. A major alternative today is networks of workstations (NOWs). Here one purchases a set of commodity PCs and networks them for use as parallel processing systems. The PCs are of course individual machines, capable of the usual uniprocessor (or now multiprocessor) applications, but by networking them together and using parallel-processing software environments, we can form very powerful parallel systems. The networking does result in a significant loss of performance. This will be discussed in Chapter \ref{chap:msg}. But even without these techniques, the price/performance ratio in NOW is much superior in many applications to that of shared-memory hardware. One factor which can be key to the success of a NOW is the use of a fast network, fast both in terms of hardware and network protocol. Ordinary Ethernet and TCP/IP are fine for the applications envisioned by the original designers of the Internet, e.g. e-mail and file transfer, but is slow in the NOW context. A good network for a NOW is, for instance, Infiniband. NOWs have become so popular that there are now ``recipes'' on how to build them for the specific purpose of parallel processing. The term {\bf Beowulf} come to mean a cluster of PCs, usually with a fast network connecting them, used for parallel processing. Software packages such as ROCKS (\url{http://www.rocksclusters.org/wordpress/}) have been developed to make it easy to set up and administer such systems. \subsection{SIMD} In contrast to MIMD systems, processors in SIMD---Single Instruction, Multiple Data---systems execute in lockstep. At any given time, all processors are executing the same machine instruction on different data. \section{Programmer World Views} To explain the two paradigms, we will use the term \textbf{nodes}, where roughly speaking one node corresponds to one processor, and use the following example: \begin{quote} Suppose we wish to multiply an nx1 vector X by an nxn matrix A, putting the product in an nx1 vector Y, and we have p processors to share the work. \end{quote} \subsection{Shared-Memory} \subsubsection{Programmer View} In the shared-memory paradigm, the arrays for A, X and Y would be held in common by all nodes. If for instance node 2 were to execute \begin{Verbatim}[fontsize=\relsize{-2}] Y[3] = 12; \end{Verbatim} and then node 15 were to subsequently execute \begin{Verbatim}[fontsize=\relsize{-2}] print("%d\n",Y[3]); \end{Verbatim} then the outputted value from the latter would be 12. \subsubsection{Example} Today, programming on shared-memory multiprocessors is typically done via \textbf{threading}. (Or, as we will see in other chapters, by higher-level code that runs threads underneath.) A \textbf{thread} is similar to a \textbf{process} in an operating system (OS), but with much less overhead. Threaded applications have become quite popular in even uniprocessor systems, and Unix,\footnote{Here and below, the term {\it Unix} includes Linux.} Windows, Python, Java and Perl all support threaded programming. In the typical implementation, a thread is a special case of an OS process. One important difference is that the various threads of a program share memory. (One can arrange for processes to share memory too in some OSs, but they don't do so by default.) On a uniprocessor system, the threads of a program take turns executing, so that there is only an illusion of parallelism. But on a multiprocessor system, one can genuinely have threads running in parallel. One of the most popular threads systems is Pthreads, whose name is short for POSIX threads. POSIX is a Unix standard, and the Pthreads system was designed to standardize threads programming on Unix. It has since been ported to other platforms. Following is an example of Pthreads programming, in which we determine the number of prime numbers in a certain range. Read the comments at the top of the file for details; the threads operations will be explained presently. \begin{Verbatim}[fontsize=\relsize{-2},numbers=left] // PrimesThreads.c // threads-based program to find the number of primes between 2 and n; // uses the Sieve of Eratosthenes, deleting all multiples of 2, all // multiples of 3, all multiples of 5, etc. // for illustration purposes only; NOT claimed to be efficient // Unix compilation: gcc -g -o primesthreads PrimesThreads.c -lpthread -lm // usage: primesthreads n num_threads #include #include #include // required for threads usage #define MAX_N 100000000 #define MAX_THREADS 25 // shared variables int nthreads, // number of threads (not counting main()) n, // range to check for primeness prime[MAX_N+1], // in the end, prime[i] = 1 if i prime, else 0 nextbase; // next sieve multiplier to be used // lock for the shared variable nextbase pthread_mutex_t nextbaselock = PTHREAD_MUTEX_INITIALIZER; // ID structs for the threads pthread_t id[MAX_THREADS]; // "crosses out" all odd multiples of k void crossout(int k) { int i; for (i = 3; i*k <= n; i += 2) { prime[i*k] = 0; } } // each thread runs this routine void *worker(int tn) // tn is the thread number (0,1,...) { int lim,base, work = 0; // amount of work done by this thread // no need to check multipliers bigger than sqrt(n) lim = sqrt(n); do { // get next sieve multiplier, avoiding duplication across threads // lock the lock pthread_mutex_lock(&nextbaselock); base = nextbase; nextbase += 2; // unlock pthread_mutex_unlock(&nextbaselock); if (base <= lim) { // don't bother crossing out if base known composite if (prime[base]) { crossout(base); work++; // log work done by this thread } } else return work; } while (1); } main(int argc, char **argv) { int nprimes, // number of primes found i,work; n = atoi(argv[1]); nthreads = atoi(argv[2]); // mark all even numbers nonprime, and the rest "prime until // shown otherwise" for (i = 3; i <= n; i++) { if (i%2 == 0) prime[i] = 0; else prime[i] = 1; } nextbase = 3; // get threads started for (i = 0; i < nthreads; i++) { // this call says to create a thread, record its ID in the array // id, and get the thread started executing the function worker(), // passing the argument i to that function pthread_create(&id[i],NULL,worker,i); } // barrier, to wait for all done for (i = 0; i < nthreads; i++) { // this call said to wait until thread number id[i] finishes // execution, and to assign the return value of that thread to our // local variable work here pthread_join(id[i],&work); printf("%d values of base done\n",work); } // report results nprimes = 1; for (i = 3; i <= n; i++) if (prime[i]) { nprimes++; } printf("the number of primes found was %d\n",nprimes); } \end{Verbatim} To make our discussion concrete, suppose we are running this program with two threads. Suppose also the both threads are running simultaneously most of the time. This will occur if they aren't competing for turns with other big threads, say if there are no other big threads, or more generally if the number of other big threads is less than or equal to the number of processors minus two. Note the global variables: \begin{Verbatim}[fontsize=\relsize{-2}] int nthreads, // number of threads (not counting main()) n, // range to check for primeness prime[MAX_N+1], // in the end, prime[i] = 1 if i prime, else 0 nextbase; // next sieve multiplier to be used pthread_mutex_t nextbaselock = PTHREAD_MUTEX_INITIALIZER; pthread_t id[MAX_THREADS]; \end{Verbatim} This will require some adjustment for those who've been taught that global variables are ``evil.'' All communication between threads is via global variables, so if they are evil, they are a necessary evil. Personally I think the stern admonitions against global variables is overblown anyway. See \url{http://heather.cs.ucdavis.edu/~matloff/globals.html}. As mentioned earlier, the globals are shared by all processors.\footnote{Technically, we should say ``shared by all threads'' here, as a given thread does not always execute on the same processor, but at any instant in time each executing thread is at some processor, so the statement is all right.} If one processor, for instance, assigns the value 0 to {\bf prime{[}35{]}} in the function {\bf crossout()}, then that variable will have the value 0 when accessed by any of the other processors as well. On the other hand, local variables have different values at each processor; for instance, the variable {\bf i} in that function has a different value at each processor. Note that in the statement \begin{Verbatim}[fontsize=\relsize{-2}] pthread_mutex_t nextbaselock = PTHREAD_MUTEX_INITIALIZER; \end{Verbatim} the right-hand side is not a constant. It is a macro call, and is thus something which is executed. In the code \begin{Verbatim}[fontsize=\relsize{-2}] pthread_mutex_lock(&nextbaselock); base = nextbase nextbase += 2 pthread_mutex_unlock(&nextbaselock); \end{Verbatim} we see a \textbf{critical section} operation which is typical in shared-memory programming. In this context here, it means that we cannot allow more than one thread to execute \begin{Verbatim}[fontsize=\relsize{-2}] base = nextbase; nextbase += 2; \end{Verbatim} at the same time. The calls to {\bf pthread\_mutex\_lock()} and {\bf pthread\_mutex\_unlock()} ensure this. If thread A is currently executing inside the critical section and thread B tries to lock the lock by calling {\bf pthread\_mutex\_lock()}, the call will block until thread B executes {\bf pthread\_mutex\_unlock()}. Here is why this is so important: Say currently {\bf nextbase} has the value 11. What we want to happen is that the next thread to read {\bf nextbase} will ``cross out'' all multiples of 11. But if we allow two threads to execute the critical section at the same time, the following may occur: \begin{itemize} \item thread A reads {\bf nextbase}, setting its value of {\bf base} to 11 \item thread B reads {\bf nextbase}, setting its value of {\bf base} to 11 \item thread A adds 2 to {\bf nextbase}, so that {\bf nextbase} becomes 13 \item thread B adds 2 to {\bf nextbase}, so that {\bf nextbase} becomes 15 \end{itemize} Two problems would then occur: \begin{itemize} \item Both threads would do ``crossing out'' of multiples of 11, duplicating work and thus slowing down execution speed. \item We will never ``cross out'' multiples of 13. \end{itemize} Thus the lock is crucial to the correct (and speedy) execution of the program. Note that these problems could occur either on a uniprocessor or multiprocessor system. In the uniprocessor case, thread A's turn might end right after it reads {\bf nextbase}, followed by a turn by B which executes that same instruction. In the multiprocessor case, A and B could literally be running simultaneously, but still with the action by B coming an instant after A. This problem frequently arises in parallel database systems. For instance, consider an airline reservation system. If a flight has only one seat left, we want to avoid giving it to two different customers who might be talking to two agents at the same time. The lines of code in which the seat is finally assigned (the \textbf{commit} phase, in database terminology) is then a critical section. A critical section is always a potential bottlement in a parallel program, because its code is serial instead of parallel. In our program here, we may get better performance by having each thread work on, say, five values of {\bf nextbase} at a time. Our line \begin{Verbatim}[fontsize=\relsize{-2}] nextbase += 2; \end{Verbatim} would become \begin{Verbatim}[fontsize=\relsize{-2}] nextbase += 10; \end{Verbatim} That would mean that any given thread would need to go through the critical section only one-fifth as often, thus greatly reducing overhead. On the other hand, near the end of the run, this may result in some threads being idle while other threads still have a lot of work to do. Note the {\bf barrier}: \begin{Verbatim}[fontsize=\relsize{-2}] for (i = 0; i < nthreads; i++) { pthread_join(id[i],&work); printf("%d values of base done\n",work); } \end{Verbatim} A barrier is a point in the code that all threads must reach before continuing. In this case, a barrier is needed in order to prevent premature execution of the later code \begin{Verbatim}[fontsize=\relsize{-2}] for (i = 3; i <= n; i++) if (prime[i]) { nprimes++; } \end{Verbatim} which would result in possibly wrong output if we start counting primes before some threads are done. The {\bf pthread\_join()} function actually causes the given thread to exit, so that we then ``join'' the thread that created it, i.e. {\bf main()}. Thus some may argue that this is not really a true barrier. Barriers are very common in shared-memory programming, and will be discussed in more detail in Chapter \ref{chap:shared}. \subsection{Message Passing} \subsubsection{Programmer View} By contrast, in the message-passing paradigm, all nodes would have \underbar{separate} copies of A, X and Y. In this case, in our example above, in order for node 2 to send this new value of Y{[}3{]} to node 15, it would have to execute some special function, which would be something like \begin{Verbatim}[fontsize=\relsize{-2}] send(15,12,"Y[3]"); \end{Verbatim} and node 15 would have to execute some kind of {\bf receive()} function. The conventional wisdom is that the shared-memory paradigm is much easier to program in than the message-passing paradigm. The latter, however, may be easier to implement, and in some settings may have greater speed. \subsection{Example} \label{mpiex} Here we use the MPI system, with our hardware being a NOW. MPI is a popular public-domain set of interface functions, callable from C/C++, to do message passing. We are again counting primes, though in this case using a \textbf{pipelining} method. It is similar to hardware pipelines, but in this case it is done in software, and each ``stage'' in the pipe is a different computer. The program is self-documenting, via the comments. \begin{Verbatim}[fontsize=\relsize{-2},numbers=left] /* MPI sample program; NOT INTENDED TO BE EFFICIENT as a prime finder, either in algorithm or implementation MPI (Message Passing Interface) is a popular package using the "message passing" paradigm for communicating between processors in parallel applications; as the name implies, processors communicate by passing messages using "send" and "receive" functions finds and reports the number of primes less than or equal to N uses a pipeline approach: node 0 looks at all the odd numbers (i.e. has already done filtering out of multiples of 2) and filters out those that are multiples of 3, passing the rest to node 1; node 1 filters out the multiples of 5, passing the rest to node 2; in this simple example, we just have node 2 filter out all the rest and then report the number of primes note that we should NOT have a node run through all numbers before passing them on to the next node, since we would then have no parallelism at all; on the other hand, passing on just one number at a time isn't efficient either, due to the high overhead of sending a message if it is a network (tens of microseconds until the first bit reaches the wire, due to software delay); thus efficiency would be greatly improved if each node saved up a chunk of numbers before passing them to the next node */ // this include file is mandatory #include #define MAX_N 100000 #define PIPE_MSG 0 // type of message containing a number to be checked #define END_MSG 1 // type of message indicating no more data will be coming int NNodes, /* number of nodes in computation*/ N, /* find all primes from 2 to N */ Me, /* my node number */ ToCheck; /* current number to check for passing on to next node; stylistically this might be nicer as a local in Node*(), but I have placed it here to dramatize the fact that the globals are NOT shared among the nodes */ double T1,T2; /* start and finish times */ Init(Argc,Argv) int Argc; char **Argv; { int DebugWait; N = atoi(Argv[1]); DebugWait = atoi(Argv[2]); /* this loop is here to synchronize all nodes for debugging; if DebugWait is specified as 1 on the command line, all nodes wait here until the debugging programmer starts GDB at all nodes and within GDB sets DebugWait to 0 to then proceed */ while (DebugWait) ; /* mandatory to begin any MPI program */ MPI_Init(&Argc,&Argv); /* puts the number of nodes in NNodes */ MPI_Comm_size(MPI_COMM_WORLD,&NNodes); /* puts the node number of this node in Me */ MPI_Comm_rank(MPI_COMM_WORLD,&Me); /* OK, get started; first record current time in T1 */ if (Me == 2) T1 = MPI_Wtime(); } Node0() { int I,Dummy, Error; /* not checked in this example */ for (I = 1; I <= N/2; I++) { ToCheck = 2 * I + 1; if (ToCheck > N) break; /* MPI_Send -- send a message parameters: pointer to place where message is to be drawn from number of items in message item type destination node message type ("tag") programmer-defined node group number (in this case all nodes) */ if (ToCheck % 3 > 0) Error = MPI_Send(&ToCheck,1,MPI_INT,1,PIPE_MSG,MPI_COMM_WORLD); } Error = MPI_Send(&Dummy,1,MPI_INT,1,END_MSG,MPI_COMM_WORLD); } Node1() { int Error, /* not checked in this example */ Dummy; MPI_Status Status; /* see below */ while (1) { /* MPI_Recv -- receive a message parameters: pointer to place to store message number of items in message (see notes on this at the end of this file) item type accept message from which node(s) message type ("tag"), programmer-defined (in this case any type) node group number (in this case all nodes) status (see notes on this at the end of this file) */ Error = MPI_Recv(&ToCheck,1,MPI_INT,0,MPI_ANY_TAG, MPI_COMM_WORLD,&Status); if (Status.MPI_TAG == END_MSG) break; if (ToCheck % 5 > 0) Error = MPI_Send(&ToCheck,1,MPI_INT,2,PIPE_MSG,MPI_COMM_WORLD); } /* now send our end-of-data signal, which is conveyed in the message type, not the message (we have a dummy message just as a placeholder */ Error = MPI_Send(&Dummy,1,MPI_INT,2,END_MSG,MPI_COMM_WORLD); } Node2() { int ToCheck, /* current number to check from Node 0 */ Error, /* not checked in this example */ PrimeCount,I,IsComposite; MPI_Status Status; /* see below */ PrimeCount = 3; /* must account for the primes 2, 3 and 5, which won't be detected below */ while (1) { Error = MPI_Recv(&ToCheck,1,MPI_INT,1,MPI_ANY_TAG, MPI_COMM_WORLD,&Status); if (Status.MPI_TAG == END_MSG) break; IsComposite = 0; for (I = 7; I*I <= ToCheck; I += 2) if (ToCheck % I == 0) { IsComposite = 1; break; } if (!IsComposite) PrimeCount++; } /* check the time again, and subtract to find run time */ T2 = MPI_Wtime(); printf("elapsed time = %f\n",(float)(T2-T1)); /* print results */ printf("number of primes = %d\n",PrimeCount); } main(argc,argv) int argc; char **argv; { Init(argc,argv); /* note: instead of having a switch statement, we could write three different programs, each running on a different node */ switch (Me) { case 0: Node0(); break; case 1: Node1(); break; case 2: Node2(); }; /* mandatory for all MPI programs */ MPI_Finalize(); } /* explanation of "number of items" and "status" arguments at the end of MPI_Recv(): when receiving a message you must anticipate the longest possible message, but the actual received message may be much shorter than this; you can call the MPI_Get_count() function on the status argument to find out how many items were actually received the status argument will be a pointer to a struct, containing the node number, message type and error status of the received message say our last parameter is Status; then Status.MPI_SOURCE will contain the number of the sending node, and Status.MPI_TAG will contain the message type; these are important if used MPI_ANY_SOURCE or MPI_ANY_TAG in our node or tag fields but still have to know who sent the message or what kind it is */ \end{Verbatim} The set of machines can be heterogeneous, but MPI ``translates'' for you automatically. If say one node has a big-endian CPU and another has a little-endian CPU, MPI will do the proper conversion.