\chapter{Recurring Performance Issues} \label{chap:issues} {\it Oh no! It's actually slower in parallel!}---almost everyone's exclamation the first time they try to parallelize code \bigskip The available parallel hardware systems sound wonderful at first. But everyone who uses such systems has had the experience of enthusiastically writing his/her first parallel program, anticipating great speedups, only to find that the parallel code actually runs more slowly than the original nonparallel program. In this chapter, we highlight some major issues that will pop up throughout the book. \section{Communication Bottlenecks} Whether you are on a shared-memory, message-passing or other platform, communication is always a potential bottleneck: \begin{itemize} \item On a shared-memory system, the threads must contend with each other in communicating with memory. And the problem is exacerbated by cache coherency transactions (Section \ref{cachecoherency}. \item On a cluster, even a very fast network is very slow compared to CPU speeds. \item GPUs are really fast, but their communication with their CPU hosts is slow. There are also memory contention issues as in ordinary shared-memory systems. \end{itemize} Among other things, communication considerations largely drive the load balancing issue, discussed next. \section{Load Balancing} \label{loadbalance} Arguably the most central performance issue is \textbf{load balancing,} i.e. keeping all the processors busy as much as possible. This issue arises constantly in any discussion of parallel processing. % In the graphics ray-tracing % application, for instance, suppose a ray is coming from the % ``northeast'' section of the image, and is reflected by a solid object. % Then the ray won't reach some of the ``southwest'' portions of the % image, which then means that the processors assigned to those portions % will not have any work to do which is associated with this ray. What we % need to do is then try to give these processors some other work to do; % the more they are idle, the slower our system will be. A nice, easily understandable example is shown in Chapter 7 of the book, {\it Multicore Application Programming: for Windows, Linux and Oracle Solaris}, Darryl Gove, 2011, Addison-Wesley. There the author shows code to compute the {\bf Mandelbrot set}, defined as follows. Start with any number c in the complex plane, and initialize z to 0. Then keep applying the transformation \begin{equation} z \leftarrow z^2 + c \end{equation} If the resulting sequence remains bounded (say after a certain number of iterations), we say that c belongs to the Mandelbrot set. Gove has a rectangular grid of points in the plane, and wants to determine whether each point is in the set or not; a simple but time-consuming computation is used for this determination.\footnote{You can download Gove's code from \url{http://blogs.sun.com/d/resource/map_src.tar.bz2}. Most relevant is {\bf listing7.64.c}.} Gove sets up two threads, one handling all the points in the left half of the grid and the other handling the right half. He finds that the latter thread is very often idle, while the former thread is usually busy---extremely poor {\bf load balance}. We'll return to this issue in Section \ref{methodabest}. \section{``Embarrassingly Parallel'' Applications} \label{embpar} The term {\bf embarrassingly parallel} is heard often in talk about parallel programming. \subsection{What People Mean by ``Embarrassingly Parallel''} Consider a matrix multiplication application, for instance, in which we compute AX for a matrix A and a vector X. One way to parallelize this problem would be to have each processor handle a group of rows of A, multiplying each by X in parallel with the other processors, which are handling other groups of rows. We call the problem embarrassingly parallel, with the word ``embarrassing'' meaning that the problem is too easy, i.e. there is no intellectual challenge involved. It is pretty obvious that the computation Y = AX can be parallelized very easily by splitting the rows of A into groups. By contrast, most parallel sorting algorithms require a great deal of interaction. For instance, consider Mergesort. It breaks the vector to be sorted into two (or more) independent parts, say the left half and right half, which are then sorted in parallel by two processes. So far, this is embarrassingly parallel, at least after the vector is broken in half. But then the two sorted halves must be merged to produce the sorted version of the original vector, and that process is {\it not} embarrassingly parallel; it can be parallelized, but in a more complex, less obvious manner. Of course, it's no shame to have an embarrassingly parallel problem! On the contrary, except for showoff academics, having an embarrassingly parallel application is a cause for celebration, as it is easy to program. In recent years, the term {\bf embarrassingly parallel} has drifted to a somewhat different meaning. Algorithms that are embarrassingly parallel in the above sense of simplicity tend to have very low communication between processes, key to good performance. That latter trait is the center of attention nowadays, so the term {\bf embarrassingly parallel} generally refers to an algorithm with low communication needs. For that reason, many people would NOT considered even our prime finder example in Section \ref{pthreadsprime} to be embarrassingly parallel. Yes, it was embarrassingly easy to write, but it has high communication costs, as both its locks and its global array are accessed quite often. On the other hand, the Mandelbrot computation described in Section \ref{loadbalance} is truly embarrassingly parallel, in both the old and new sense of the term. There the author Gove just assigned the points on the left to one thread and the rest to the other thread---very simple---and there was no communication between them. \subsection{Iterative Algorithms} Many parallel algorithms involve iteration, with a rendezvous of the tasks after each iteration. Within each iteration, the nodes act entirely independently of each other, which makes the problem seem embarrassingly parallel. But unless the granularity of the problem is coarse, i.e. there is a large amount of work to do in each iteration, the communication overhead will be significant, and the algorithm may not be considered embarrassingly parallel. \section{Static (But Possibly Random) Task Assignment Typically Better Than Dynamic} \label{methodabest} Say an algorithm generates t independent\footnote{Note the qualifying term.} tasks and we have p processors to handle them. In our matrix-times-vector example of Section \ref{matvec}, say, each row of the matrix might be considered one task. A processor's work would then be to multiply the vector by this processor's assigned rows of the matrix. How do we decide which tasks should be done by which processors? In {\bf static} assignment, our code would decide at the outset which processors will handle which tasks. The alternative, {\bf dynamic} assignment, would have processors determine their tasks as the computation proceeds. In the matrix-times-vector example, say we have 10000 rows and 10 processors. In static task assignment, we could pre-assign processor 0 rows 0-999, processor 1 rows 1000-1999 and so on. On the other hand, we could set up a {\bf task farm}, a queue consisting here of the numbers 0-9999. Each time a processor finished handling one row, it would remove the number at the head of the queue, and then process the row with that index. It would at first seem that dynamic assignment is more efficient, as it is more flexible. However, accessing the task farm, for instance, entails communication costs, which might be very heavy. In this section, we will show that it's typically better to use the static approach, though possibly randomized.\footnote{This is still static, as the randomization is done at the outset, before starting computation.} \subsection{Example: Matrix-Vector Multiply} Consider again the problem of multiplying a vector X by a large matrix A, yielding a vector Y. Say A has 10000 rows and we have 10 threads. Let's look at little closer at the static/dynamic tradeoff outlined above. For concreteness, assume the shared-memory setting. There are several possibilities here: \begin{itemize} \item {\bf Method A:} We could simply divide the 10000 rows into chunks of 10000/10 = 1000, and parcel them out to the threads. We would pre-assign thread 0 to work on rows 0-999 of A, thread 1 to work on rows 1000-1999 and so on. This is essentially OpenMP's {\bf static} scheduling policy, with default chunk size.\footnote{See Section \ref{schedulework}.} There would be no communication between the threads this way, but there could be a problem of load imbalance. Say for instance that by chance thread 3 finishes well before the others. Then it will be idle, as all the work had been pre-allocated. \item {\bf Method B:} OpenMP's {\bf dynamic} policy does what the name implies, which can be described as follows (in non-OpenMP terms): We would have a shared variable named,say, {\bf nextchunk} similar to {\bf nextbase} in our prime-finding program in Section \ref{pthreadsprime}. Each time a thread would finish a chunk, it would obtain a new chunk to work on, by recording the value of {\bf nextchunk} and incrementing that variable by 1 (all atomically, of course). This approach would have better load balance, because the first thread to find there is no work left to do would be idle for at most 100 rows' amount of computation time, rather than 1000 as above. Meanwhile, though, communication would increase, as access to the locks around {\bf nextchunk} would often make one thread wait for another.\footnote{Why are we calling it ``communication'' here? Recall that in shared-memory programming, the threads communicate through shared variables. When one thread increments {\bf nextchunk}, it ``communicates'' that new value to the other threads by placing it in shared memory where they will see it, and as noted earlier contention among threads to shared memory is a major source of potential slowdown.} \item {\bf Method C:} So, Method A above minimizes communication at the possible expense of load balance, while the Method B does the opposite. OpenMP also offers the {\bf guided} policy, which is like {\bf dynamic} except the chunk size decreases over time. In the earlier part of a run, the large chunk size reduces communication, while later the small chunk size avoid load imbalance. \end{itemize} I will now show that in typical settings, the Method A above (or a slight modification) works well. To this end, consider a chunk consisting of m tasks, such as m rows in our matrix example above, with times $T_1, T_2,..., T_m$. The total time needed to process the chunk is then $T_1+...,T_m$. The $T_i$ can be considered random variables; some tasks take a long time to perform, some take a short time, and so on. As an idealized model, let's treat them as independent and identically distributed random variables. Under that assumption (if you don't have the probability background, follow as best you can), we have that the mean ({\bf expected value}) and variance of total task time are $$ E(T_1+...,T_m) = m E(T_1) $$ and $$ Var(T_1+...,T_m) = m Var(T_1) $$ Thus $$ \frac {\textrm{standard deviation of chunk time}} {\textrm{mean of chunk time}} ~ \sim ~ O \left (\frac{1}{\sqrt{m}} \right ) $$ In other words: \begin{itemize} \item run time for a chunk is essentially constant if m is large, and \item there is essentially no load imbalance in Method A \end{itemize} Since load imbalance was the only drawback to Method A and we now see it's not a problem after all, then Method A is best. For more details and timing examples, see N. Matloff, ``Efficient Parallel R Loops on Long-Latency Platforms,'' {\it Proceedings of the 42nd Interface between Statistics and Computer Science}, Rice University, June 2012.\footnote{As noted in the Preface to this book, I occasionally refer here to my research, to illustrate for students the beneficial interaction between teaching and research.} \subsection{Load Balance, Revisited} But what about the assumptions behind that reasoning? Consider for example the Mandelbrot problem in Section \ref{loadbalance}. There were two threads, thus two chunks, with the tasks for a given chunk being computations for all the points in the chunk's assigned region of the picture. Gove noted there was fairly strong load imbalance here, and that the {\it reason} was that most of the Mandelbrot points turned out to be in the left half of the picture! The computation for a given point is iterative, and if a point is not in the set, it tends to take only a few iterations to discover this. That's why the thread handling the right half of the picture was idle so often. So Method A would not work well here, and upon reflection one can see that the problem was that the tasks within a chunk were not independent, but were instead highly correlated, thus violating our mathematical assumptions above. Of course, before doing the computation, Gove didn't know that it would turn out that most of the set would be in the left half of the picture. But, one could certainly anticipate the correlated nature of the points; if one point is not in the Mandelbrot set, its near neighbors are probably not in it either. But Method A can still be made to work well, via a simple modification: Simply form the chunks randomly. In the matrix-multiply example above, with 10000 rows and chunk size 1000, do NOT assign the chunks contiguously. Instead, generate a random permutation of the numbers 0,1,...,9999, naming them $i_0, i_1,...,i_{9999}$. Then assign thread 0 rows $i_0-i_{999}$, thread 1 rows $i_{1000}-i_{1999}$, etc. In the Mandelbrot example, we could randomly assign rows of the picture, in the same way, and avoid load imbalance. So, actually, Method A, or let's call it Method A', will still typically work well. \subsection{Example: Mutual Web Outlinks} \label{mutlinks} Here's an example that we'll use at various points in this book: \begin{quote} {\bf Mutual outlinks in a graph:} Consider a network graph of some kind, such as Web links. For any two vertices, say any two Web sites, we might be interested in mutual outlinks, i.e. outbound links that are common to two Web sites. Say we want to find the number of mutual outlinks, averaged over all pairs of Web sites. Let A be the {\bf adjacency matrix} of the graph. Then the mean of interest would be found as follows: \begin{lstlisting}[numbers=left] sum = 0 for i = 0...n-2 for j = i+1...n-1 count = 0 for k = 0...n-1 count += a[i][k] * a[j][k] mean = sum / (n*(n-1)/2) \end{lstlisting} Say again n = 10000 and we have 10 threads. We should not simply assign work to the threads by dividing up the i loop, with thread 0 taking the cases i = 0,...,999, thread 1 the cases 1000,...,1999 and so on. This would give us a real load balance problem. Thread 8 would have much less work to do than thread 3, say. We could randomize as discussed earlier, but there is a much better solution: Just \underline{pair} the rows of A. Thread 0 would handle rows 0,...,499 and 9500,...,9999, thread 1 would handle rows 500,999 and 9000,...,9499 etc. This approach is taken in our OpenMP implementation, Section \ref{ompmutlinks}. In other words, Method A still works well. \end{quote} In the mutual outlinks problem, we have a good idea beforehand as to how much time each task needs, but this may not be true in general. An alternative would be to do {\it random} pre-assignment of tasks to processors. On the other hand, if we know beforehand that all of the tasks should take about the same time, we should use static scheduling, as it might yield better cache and virtual memory performance. \subsection{Work Stealing} There is another variation to Method A that is of interest today, called {\bf work stealing}. Here a thread that finishes its assigned work and has thus no work left to do will ``raid'' the work queue of some other thread. This is the approach taken, for example, by the elegant Cilk language. Needless to say, accessing the other work queue is going to be expensive in terms of time and memory contention overhead. \subsection{Timing Example} \label{mandelbrottiming} I ran the Mandelbrot example on a shared memory machine with four cores, two threads per core, with the following results for eight threads, on an 8000x8000 grid: \begin{tabular}{|r|r|} \hline policy & time \\ \hline \hline static & 47.8 \\ \hline dynamic & 21.4 \\ \hline guided & 29.6 \\ \hline random & 15.7 \\ \hline \end{tabular} Default values were used for chunk size in the first three cases. I did try other chunk sizes for the {\bf dynamic} policy, but it didn't make much difference. See Section \ref{mandelbrotcode} for the code. Needless to say, one shouldn't overly extrapolate from the above timings, but it does illustrate the issues. \subsection{Example: Extracting a Submatrix} OpenMP implements the various scheduling algorithms via its {\bf for} pragma, e.g. \begin{lstlisting} #pragma omp for schedule(guided) \end{lstlisting} We'll cover this in more detail in Chapter \ref{chap:omp}, but here is an example of the default, static scheduling with chunk size 1. \begin{lstlisting} // creates a new matrix from a group of {\bf nColsSubM} // consecutive columns of an old one, starting at column // firstCol} of the old matrix #include #include #include #include int onedim(int row, int col, int nCols) { return row * nCols + col; } int *getCols(int *m, int nRowsM, int nColsM, int firstCol, int nColsSubM) { int *subMat; #pragma omp parallel { int *startFrom,*startTo; #pragma omp single subMat = malloc(nRowsM * nColsSubM * sizeof(int)); #pragma omp for for (int row = 0; row < nRowsM; row++) { startFrom = m + onedim(row,firstCol,nColsM); startTo = subMat + onedim(row,0,nColsSubM); memcpy(startTo, startFrom, nColsSubM * sizeof(int)); } } return subMat; } int main() { // intended as a 4x3 matrix int z[12] = {5,12,13,6,2,22,15,3,1,2,3,4}; omp_set_num_threads(2); // extract the last 2 cols; should be // 12 13 // 2 22 // 3 1 // 3 4 int *outM = getCols(z,4,3,1,2); for (int i = 0; i < 4; i++) { printf("%d %d\n",outM[2*i],outM[2*i+1]); } } \end{lstlisting} \section{Latency and Bandwidth} \label{latencybandwidth} We've been speaking of communications delays so far as being monolithic, but they are actually (at least) two-dimensional. The key measures are {\bf latency} and {\bf bandwidth}: \begin{itemize} \item Latency is the time it takes for one bit to travel for source to destination, e.g. from a CPU to memory in a shared memory system, or from one computer to another in a cluster. \item Bandwidth is the number of bits per unit time that can be input into the communications channel. This can be affected by factors such as bus width in a shared memory system and number of parallel network paths in a message passing system, and also by the speed of the links. \end{itemize} It's helpful to think of a bridge, with toll booths at its entrance. Latency is the time needed for one car to get from one end of the bridge to the other. Bandwidth is the number of cars that can enter the bridge per unit time. We can reduce latency by increasing the speed limit, and can increase bandwidth by improving the speed by which toll takers can collect tolls, and increasing the number of toll booths. {\bf Latency hiding:} One way of dealing with long latencies is known as {\bf latency hiding}. The idea is to do a long-latency operation in parallel with something else. For example, GPUs tend to have very long memory access times, but this is solved by having many pending memory accesses at the same time. During the latency of some accesses, earlier ones that have now completed can now be acted upon (Section \ref{coal}). \section{Relative Merits: Performance of Shared-Memory Vs. Message-Passing} \label{sharedbetter} My own preference is shared-memory, but there are pros and cons to each paradigm. It is generally believed in the parallel processing community that the shared-memory paradigm produces code that is easier to write, debug and maintain than message-passing. See for instance R. Chandra, {\it Parallel Programming in OpenMP}, MKP, 2001, pp.10ff (especially Table 1.1), and M. Hess {\it et al}, Experiences Using OpenMP Based on Compiler Directive Software DSM on a PC Cluster, in {\it OpenMP Shared Memory Parallel Programming: International Workshop on OpenMP Applications and Tools}, Michael Voss (ed.), Springer, 2003, p.216. On the other hand, in some cases message-passing can produce faster code. Consider the Odd/Even Transposition Sort algorithm, for instance. Here pairs of processes repeatedly swap sorted arrays with each other. In a shared-memory setting, this might produce a bottleneck at the shared memory, slowing down the code. Of course, the obvious solution is that if you are using a shared-memory machine, you should just choose some other sorting algorithm, one tailored to the shared-memory setting. There used to be a belief that message-passing was more {\bf scalable}, i.e. amenable to very large systems. However, GPU has demonstrated that one can achieve extremely good scalability with shared-memory. As will be seen, though, GPU is hardly a panacea. Where, then, are people to get access to large-scale parallel systems? Most people do not (currently) have access to large-scale multicore machines, while most do have access to large-scale message-passing machines, say in cloud computing venues. Thus message-passing plays a role even for those of us who preferred the shared-memory paradigm. Also, hybrid systems are common, in which a number of shared-memory systems are tied together by, say, MPI. \section{Memory Allocation Issues} Many algorithms require large amounts of memory for intermediate storage of data. It may be prohibitive to allocate this memory statically, i.e. at compile time. Yet dynamic allocation, say via {\bf malloc()} or C++'s {\bf new} (which probably produces a call to {\bf malloc() anyway}, is very expensive in time. Using large amounts of memory also can be a major source of overhead due to cache misses and page faults. One way to avoid {\bf malloc()}, of course, is to set up static arrays whenever possible. There are no magic solutions here. One must simply be aware of the problem, and tweak one's code accordingly, say by adjusting calls to {\bf malloc()} so that one achieves a balance between allocating too much memory and making too many calls. \section{Issues Particular to Shared-Memory Systems} This topic is covered in detail in Chapter \ref{chap:shared}, but is so important that the main points should be mentioned here. \begin{itemize} \item Memory is typically divided into {\bf banks}. If more than one thread attempts to access the same bank at the same time, that effectively serializes the program. \item There is typically a cache at each processor. Keeping the contents of these caches consistent with each other, and with the memory itself, adds a lot of overhead, causing slowdown. \end{itemize} In both cases, awareness of these issues should impact how you write your code. See Sections \ref{banks} and \ref{cacheissues}.