\chapter{MapReduce Computation} \label{chap:mapreduce} As the world emerged into an era of Big Data, demand grew for a computing paradigm that (a) is generally applicable and (b) works on {\it distributed} data.The latter term means that data is physically distributed over many chunks, possibly on different disks and maybe even different geographical locations. Having the data stored in a distributed manner facilitates parallel computation --- different chunks can be read simultaneously --- and also enables us to work with data sets that are too large to fit into the memory of a single machine. Demand for such computational capability led to the development of various systems using the {\it MapReduce} paradigm. MapReduce is really a form of the scatter-gather pattern we've seen frequently in this book, with the added feature of a sorting operation added in the middle. In rough form, it works like this. The input is in a (distributed) file, fed into the following process: \begin{itemize} \item {\bf Map phase:} There are various parallel processes known as {\it mappers}, all running the same code. For each line of the input file, the mapper handling that chunk of the file reads the line, processes it in some way, and then emits an output line, consisting of a key-value pair. \item {\bf Shuffle/sort phase:} All the mapper output lines that share the same key are gathered together. \item {\bf Reduce phase:} There are various parallel processes known as {\it reducers}, all running the same code. Each reducer will work on its own set of keys, i.e.\ for any given key, all mapper output lines having the same key will go to the same reducer. Moreover, the lines fed into any given reducer will be sorted by key. \end{itemize} \section{Apache Hadoop} At this writing, the most popular MapReduce software package is Hadoop. It is written in Java, and is most efficiently used in that language (or C++), but it includes a {\it streaming} feature that enables one to use Hadoop from essentially any language, including R. Hadoop includes is into distributed file system, unsurprisingly called the Hadoop Distributed File System (HDFS). Note that one advantage of HDFS is that it is replicated, thus achieving some degree of fault tolerance. \subsection{Hadoop Streaming} As noted, Hadoop is really written for Java or C++ applications. However, Hadoop can work with programs in any language under Hadoop's streaming option, by reading from {\bf stdin} and writing to {\bf stdout}, in text, line-oriented form in both cases. Input to the mappers is from an HDFS file, and output from the reducers is again to an HDFS file, one chunk per reducer. The file line format is \begin{verbatim} key \t data \end{verbatim} where \lstinline{\t} is the Tab character. The usage of text format does cause some slowdown in numeric programs, for the conversion of strings to numbers and vice versa, but again, Hadoop is not designed for maximum efficiency. \subsection{Example: Word Count} \label{hadoopwordcount} The typical ``Hello World,'' introductory example is word count for a text file. The mapper program breaks a line into words, and emits (key,value) pairs in the form of (word,1). (If a word appears several times in a line, there would be several pairs emitted for that word.) In the Reduce stage, all those 1s for a given word are summed, giving a frequency count for that word. In this way, we get counts for all words. Here is the mapper code: \begin{lstlisting} #! /usr/bin/env Rscript # wordmapper.R si <- file("stdin",open="r") while (length(inln <- scan(si,what="",nlines=1,quiet=TRUE,blank.lines.skip=FALSE))) { for (w in inln) { cat(w,"\t 1\n") } } \end{lstlisting} And here is the reducer: \begin{lstlisting} #! /usr/bin/env Rscript # wordreducer.R si <- file("stdin",open="r") oldword <- "" while (length(inln <- scan(si,what="",nlines=1,quiet=TRUE))) { word <- inln[1] if (word != oldword) { if (oldword != "") cat(oldword,"\t ",count,"\n") oldword <- word count <- 1 } else { count <- count + as.integer(inln[2]) } } \end{lstlisting} The above code is not very refined, for instance treating {\it The} as different from {\it the}. The main point, though, is just to illustrate the principles. \subsection{Running the Code} I ran the following code from the top level of the Hadoop directory tree, with obvious modifications possible for other run points. First, I needed to place my data file, {\bf rnyt},\footnote{The file was the contents of the article ``Data Analysts Captivated by R’s Power,'' {\it New York Times}, January 6, 2009.} \begin{lstlisting} $ bin/hadoop fs -put ../rnyt rnyt $ bin/hadoop jar contrib/streaming/*.jar \ -input rnyt \ -output wordcountsnyt \ -mapper ../wordmapper.R -reducer ../wordreducer.R \end{lstlisting} That first command specifies that it will be for the file system ({\bf fs}, and that I am placing a file in that system. My ordinary version of the file was in my home directory. Hadoop, being Java-based, runs Java archive, {\bf .jar} files. So, the second command above specifies that I want to run in streaming mode. It also states that I wan the output to go to a file {\bf wordcountsnyt} in my HDFS system. Finally, I specify my mapper and reducer code files. in my HDFS system, after which I ran the program. I allowed Hadoop to use its default values for the number of mappers and reducers, and could have specified them above if desired. Recall that the final output comes in chunks in the HDFS. Here's how to check (some material not shown), and to view the actual file contents: \begin{lstlisting} $ bin/hadoop fs -ls wordcountsnyt Found 3 items -rw-r--r-- 1 ... /user/matloff/wordcountsnyt/_SUCCESS drwxr-xr-x - ... /user/matloff/wordcountsnyt/_logs -rw-r--r-- 1 ... /user/matloff/wordcountsnyt/part-00000 $ bin/hadoop fs -cat wordcountsnyt1/part-00000 1 NA $2 1 1,600 1 18th 1 1991, 1 1996, 1 2009 1 2009, 1 250,000 1 6, 1 7, 1 A 1 ASHLEE 1 According 1 America, 1 Analysts 2 Anne 1 Another 1 Apache, 1 Are 1 At 1 ... \end{lstlisting} So, in HDFS, one distributed file is stored as a directory, with the chunks in {\bf part-00000}, {\bf part-00001} and so on. We only had enough data to fill one chunk here. \subsection{Analysis of the Code} The first thing to notice is that these two R files are not executed directly by R, but instead under Rscript. This is standard for running R in {\it batch}, i.e.\ noninteractive, mode. Next, as noted earlier, input to the mappers is from {\bf stdin}, in this case from the redirected file {\bf rnyt} in my HDFS, seen here in the identifier {\bf si} in the mapper. Final output is to {\bf stdout}, redirected to the specified HDFS file, hence the call to {\bf cat()} in the reducer. The mapper output goes to the shuffle and then to the reducers, again using STDOUT, visible here in the call to {\bf cat()} in the mapper code. The reader might be wondering here about the line \begin{lstlisting} count <- count + as.integer(inln[2]) \end{lstlisting} in the reducer. The way things have been described so far, it would seem that the expression {\bf as.integer(inln[2])} --- a word count output from a mapper --- should always be 1. However, there is more to the story, as Hadoop also allows one to specify {\it combiner} code, as follows. Remember, all the communication, e.g.\ from the mappers to the shuffler, is via our network, with one (key,value) pair per network message. So, we may have an enormous number of short messages, thus incurring the network latency penalty many times, as well as huge network congestion due to, for instance, many mappers trying to use the network at once. The solution is to have each mapper try to coalesce its messages before sending to the shuffler. The coalescing is done by a {\it combiner} specified by the user. Often the combiner will be the same as the reducer. So, what occurs is that each mapper will run the reducer on its own mapper output, then send the combiner output to the shuffler, after which it goes to the reducers as usual. Thus in our word count example here, when a line arrives at a reducer, its count field may already have a value greater than 1. The combiner code, by the way, is specified via the {\bf -combiner} field in the run command, like {\bf -mapper} and {\bf -reducer}. \subsection{Role of Disk Files} As ntoed, Hadoop has its own file system, HDFS, which is built on top of the native OS' file system of the machines. It is replicated for the sake of reliability, with each HDFS block existing in at least 3 copies, i.e. on at least 3 separate disks. Very large files are possible, in some cases spanning more than one disk/machine. Disk files play a major role in Hadoop programs: \begin{itemize} \item Input is from a file in the HDFS system. \item The output of the mappers goes to temporary files in the native OS' file system. \item Final output is to a file in the HDFS system. As noted earlier, that file may be distributed across several disks/machines. \end{itemize} Note that by having the input and output files in HDFS, we minimize communications costs in shipping the data between nodes of a cluster. The slogan used is ``Moving computation is cheaper than moving data.'' However, all that disk activity can be quite costly in terms of run time. \section{Other MapReduce Systems} As of late 2014, there has been increasing concern regarding Hadoop's performance. One of the problems is that one cannot keep intermediate results in memory between Hadoop runs. This is a serious problem, for instance, with iterative or even multi-pass algorithms. The Spark package now being developed aims to remedy many of Hadoop's shortcomings. Early reports indicate some drastic speed improvements, while retaining the ability to read HDFS files, and continuing to have fault tolerance features. \section{R Interfaces to MapReduce Systems} Given the widespread usage of Hadoop, a number of R interfaces have been developed. The most popular is probably {\bf rmr}, developed by Revolution Analytics, and RHIPE, written by Saptarshi Guha as part of his PhD dissertation. An R interface package, {\bf sparkr}, is available. \section{An Alternative: ``Snowdoop''} \label{snowdoop} So, what does Hadoop really give us? The two main features are (a) distributed data access and (b) an efficient distributed file sort. Hadoop works well for many applications, but a realization developed that Hadoop can be very slow, and very limited in available data operations. Both of those shortcomings are addressed to a large extent by the new kid on the block, Spark. Spark is apparently much faster than Hadoop, sometimes dramatically so, due to strong caching ability and a wider variety of available operations. Recently {\bf distributedR} has also been released, again with the goal of using R on voluminous data sets, and there is also the more established {\bf pbdR}. But even Spark suffers a very practical problem, shared by the others mentioned above. All of these systems are complicated. There is a considerable amount of configuration to do, worsened by dependence on infrastructure software such as Java or MPI, and in some cases by interface software such as rJava. Some of this requires systems knowledge that many R users may lack. And once they do get these systems set up, they may be required to design algorithms with world views quite different from R, even though they are coding in R. So, do we really need all that complicated machinery? Hadoop and Spark provide efficient distributed sort operations, but if one's application does not depend on sorting, we have a cost-benefit issue here. Here is an alternative, a general approach rather than a package, which I call ``Snowdoop'': One simply does one's own chunking of files into distributed mini-files, and then uses Snow or some other general R tool on those files. \subsection{Example: Snowdoop Word Count} Let's use as our example word count, the ``Hello World'' of MapReduce. It determines which words are in a text file, and calculates frequency counts for each distinct word: \begin{lstlisting}[numbers=left] # each node executes this function wordcensus <- function(basename,ndigs) { fname <- filechunkname(basename,ndigs) words <- scan(fname,what="") tapply(words,words,length, simplify=FALSE) } # manager fullwordcount <- function(cls,basename,ndigs) { setclsinfo(cls) # give workers ID numbers, etc. counts <- clusterCall(cls,wordcensus,basename,ndigs) addlistssum <- function(lst1,lst2) addlists(lst1,lst2,sum) Reduce(addlistssum,counts) } \end{lstlisting} The above code makes use of the following routines, which are general and are used in many ``Snowdoop'' applications. These and other utilities are included in my {\bf partools} package. Here are the call forms: \begin{lstlisting} # give each cluster node an ID, etc. setclsinfo(cls) # "add" lists lst1, lst2, applying the operation 'add' to elements in # common, copying non-null others addlists(lst1,lst2,add) # form the file name basename.i, where i is the ID of this node unless # nodenum is specified filechunkname(basename,ndigs,nodenum=NULL) \end{lstlisting} All pure R! No Java, no configuration. Indeed, it's worthwhile comparing to the word count example in the {\bf sparkr} distribution. There we see calls to {\bf sparkr} functions such as {\bf flatMap()}, {\bf reduceByKey()} and {\bf collect()}. But the {\bf reduceByKey()} function is pretty much the same as R's tried and true {\bf tapply()}. The {\bf collect()} function is more or less our Snowdoop library function {\bf addlists()}. So, again, there is no need to resort to Spark, Hadoop, Java and so on; we just use ordinary R. We are achieving the parallel-read advantage of Hadoop and Spark,\footnote{Note that neither Hadoop, Spark nor Snowdoop will achieve full parallel reading if the file chunks are all on the same disk.} {\it while avoiding the Hadoop/Spark configuration headaches and while staying with the familiar R programming paradigm}. In many cases, this should be a highly beneficial tradeoff for us. Of course, this approach lacks the fault tolerance feature of Hadhoop and Spark can, which can be quite advantageous. And as of this writing, it is not yet clear how well this scales, e.g.\ how well the {\bf parallel} package works with very large numbers of nodes. But Snowdoop is an attractive approach for many applications. \subsection{Example: Snowdoop k-Means Clustering} The k-means clustering problem is discussed in Section \ref{cluster}. Read ahead, and then see how we can implement it here: \begin{lstlisting}[numbers=left] # k-means clustering, using Snowdoop # chunked data with name xname, nitrs iterations, nclus clusters; # assumes for simplicity that a cluster will never become empty; ctrs is # the matrix of initial centroids # assumes setclsinfo already called kmeans <- function(cls,xname,nitrs,ctrs) { # will tell everyone to read their chunks; first, find cluster size # and compute number of digits in file suffixes addlistssum <- function(lst1,lst2) addlists(lst1,lst2,sum) for (i in 1:nitrs) { # for each data point, find the nearest centroid, and tabulate; at # each worker and for each centroid, we compute a vector whose # first component is the count of the number of data points whose # nearest centroid is that centroid, and whose remainining portion # is the sum of all such data points tmp <- clusterCall(cls,findnrst,xname,ctrs) # sum over all workers tmp <- Reduce(addlistssum,tmp) # compute new centroids for (i in 1:nrow(ctrs)) { tmp1 <- tmp[[as.character(i)]] ctrs[i,] <- (1/tmp1[1]) * tmp1[-1] } } ctrs } findnrst <- function(xname,ctrs) { require(pdist) x <- get(xname) dsts <- matrix(pdist(x,ctrs)@dist,ncol=nrow(x)) # dsts[,i] now has the distances from row i of x to the centroids nrst <- apply(dsts,2,which.min) # nrst[i] tells us the index of the centroid closest to row i of x mysum <- function(idxs,myx) c(length(idxs),colSums(x[idxs,,drop=F])) tmp <- tapply(1:nrow(x),nrst,mysum,x) } test <- function(cls) { m <- matrix(c(4,1,4,6,3,2,6,6),ncol=2) formrowchunks(cls,m,"m") initc <- rbind(c(2,2),c(3,5)) kmeans(cls,"m",1,initc) } \end{lstlisting} So again, we are using chunked files as in Hadoop, but writing ordinary R code, e.g. {\bf tapply()} and {\bf Reduce()}. But most important, the data at each worker persists across iterations. In Hadoop, it would be reread from disk at each iteration, and in Spark, we'd need to request caching, but here it comes for free, no special effort needed.