\chapter{Some Infrastructure: Probability and Statistics} \label{chap:infra2} Many RS methods are probabilistic in nature, so we will lay out some infrastructure. It is assumed the reader has a background in calculus-based probability structures, such as expected value and density functions. Background in statistics (as opposed to probability) and machine learning is \textit{not} assumed.\footnote{The reader may wish to consult my open source book on probability and statistics, N. Matloff, \textit{From Algorithms to Z-Scores: Probability and Statistical Modeling in Computer Science}, \url{http://heather.cs.ucdavis.edu/probstatbook}.} Note that while we will develop some statistical methods here, notably regression and classification models, we will not cover \textit{inferential} statistical methods such as confidence intervals and significance tests. For readers familiar with such topics, occasional footnotes will be provided. Other than laying some groundwork, e.g. introducing the term \textit{dummy variables}, the primary goal of this chapter will be to discuss the issue of \textit{overfitting}. \section{Data as a Sample} In statistics, the data are usually considered a sample from a population. For instance, during an election campaign pollsters will take a \underline{sample}, typically of 1200 people, from the \underline{population} of all voters. Say they are interested in $p$, the population proportion of voters favoring Candidate Jones. They calculate their \underline{estimate} of $p$, denoted $\widehat{p}$, to be the proportion of voters in the \underline{sample} who like Jones. Sometimes the notion of sampling is merely conceptual. Say for instance we are studying hypertension, on data involving 1000 patients. We think of them as a sample from the population of all sufferers of hypertension, even though we did not go through an actual sampling process. In RS contexts, this means that we treat the users in our dataset as a sample from the conceptual population of all potential users. We may even treat the items as a sample from a conceptual population of all possible items. In machine learning circles, it is not customary to think explicitly in terms of populations, samples and estimates. Nevertheless, it's implied, as ML people do talk about predicting new data from the model fitted on the original data. For the model to be useful, the new data needs to come from the same source as the original --- what statisticians call a population. We will usually think in terms of sample data here. \section{Probability, Expected Value and Variance} We will speak in terms of a \textit{repeatable experiment}, which again could be physical or conceptual. We think of probability as the long-run proportion of the time some event occurs. Say we toss a fair coin. What do we mean by $P(\textrm{heads} = 0.5)$? Here our repeatable experiment is tossing the coin. If we were to perform that experiment many times --- ideally, infinitely many times --- then in the long run, 50\% of the repetitions would give us heads. Now suppose our experiment, say a game, is to keep tossing a coin until we get three consecutive heads. Let $X$ denote the number of tosses needed. Then for instance $P(X = 4) = 0.5^4 = 0.0625$ (we get a tail then three heads). Imagine doing this experiment infinitely many times: We toss the coin until we get three consecutive heads, and record $X$; we toss the coin until we get three consecutive heads, and record $X$; we toss the coin until we get three consecutive heads, and record $X$; and so on. This would result in infinitely many $X$ values. Then in the long run, 6.25\% of the $X$ values would be 4. The \textit{expected value} $E(X)$ of a random variable $X$ is its long-run average value over infinitely many repetitions of the experiment. In that 3-consecutive heads game above, it can be shown that $E(X) = 14.7$. In other words, if we were to play the game infinitely many times, yielding infinitely $X$ values, their long-run average would be 14.7. If there is no danger of ambiguity, we usually omit the parentheses, writing $EX$ instead of $E(X)$. The \textit{variance} of a random variable is a measure of its dispersion, i.e.\ how much it varies from one repetition to the next. It is defined as $Var(X) = E[(X - EX)^2]$. Say we have a population of people and our our experiment is to randomly draw one person from the population, denoting that person's height by $H$. Then intuitively, $EH$ will be the mean height of all the people in the population, traditionally written as $\mu$. \section{Regression Models} Regression models, both \textit{parametric} and \textit{nonparametric}, \textbf{form the very core of statistics and machine learning (ML)}. Their importance cannot be overemphasized.\footnote{For many, the term \textit{regression analysis} connotes a linear parametric model. But actually the term is much more general, defined to be the conditional mean as discussed below. Most ML techniques are nonparametric, as explained below, but are still regression methods.} \subsection{Definition} Suppose we are predicting a variable $Y$ from a vector $X$ of variables, say predicting human weight from the vector (height,age). The \textit{regression function} at $t = (t_1,t_2)$ of $Y$ on $X$ is defined to be the mean weight of all people in the subpopulation consisting of all people of height $t_1$ and age $t_2$. Let $W$, $H$ and $A$ denote weight, height and age. We write the regression function as the \textit{conditional expectation} of $W$ given $H$ and $A$, \begin{equation} \label{regdef} E(W ~| H=t_1, A=t_2) \end{equation} If, say $E(W ~| H=70, A=28) = 162$, it means that the mean weight of all people in the subpopulation consisting of 28-year-olds of height 70 is 162. Note that in (\ref{regdef}), the expression has a different value for each $(t_1,t_2)$ pair. So it is a \underline{function} of $t_1$ and $t_2$. This is why it is called the \textit{regression function} of $W$ on $H$ and $A$. \textit{Terminology:} It is common to refer to $W$ here at the \textit{response variable} and $H$ and $A$ as the \textit{predictor variables}. The latter may also be called \textit{explanatory variables} (in economics and other social sciences) or \textit{features} (in ML). \subsection{Prediction} Say we have a person whose height and age are 70 and 28, but with unknown weight. It can be shown that the optimal (under a certain criterion) predictor of her weight is the value of the regression function at (70,28), $E(W ~|~ H=70, A=28) = 162$. It is optimal in the sense of minimizing expected squared prediction error. \subsection{Estimation} The regression function is an attribute of the population. Yet all we have is sample data. How do we estimate the regression function from our data? \subsubsection{Nonparametric} Intuitively, we could use a nearest-neighbor approach. To estimate $E(W ~| H=70, A=28)$, we could find, say, the 25 people in our sample for whom $(H,A)$ is closest to (70,28), and average their weights to produce our estimate of $E(W ~| H=70, A=28)$. This kind of approach is common in ML. The famous \textit{random forests} method is basically a more complex form of kNN, as we will see in Chapter \ref{chap:covars}. Statisticians also use methods like kNN. In fact, kNN and random forests were invented by statisticians. But more commonly, statistics uses \textit{parametric} methods, as follows. \subsubsection{Parametric} The basic idea is to assume the regression function is linear in \underline{parameters} $\beta_i$, e.g. \begin{equation} \label{wthtage} \textrm{mean weight} = \beta_0 + \beta_1 ~ \textrm{height} + \beta_2 ~ \textrm{age} \end{equation} for some unknown values of the $\beta_i$. Make sure to take careful note of the word ``mean''! Clearly, the weights of \underline{individual} people are not linear functions of their height and age. As noted, the $\beta_i$ are unknown, and need to be estimated from our sample data. The estimates will be denoted $\widehat{\beta}_i$. They are obtained by minimizing a certain sum of squares, to be discussed in Section \ref{lmdetails}. By the way, if the reader is familiar with the ML methodology known as \textit{neural networks}, she may be surprised that this technique is also parametric. Again, more in Chapter \ref{chap:covars}. \subsubsection{Comparison} Consider (\ref{wthtage}), our model for the function of $t_1$ and $t_2$ \begin{equation} E(\textrm{weight} ~|~ \textrm{height} = t_1, \textrm{age} = t_2) \end{equation} With the linear assumption (\ref{wthtage}), we will be estimating three quantities, the $\beta_i$. But with a nonparametric approach, we are estimating infinitely many quantities, one for each value of the $(t_1,t_2)$ pair. In other words, \textbf{parametric methods are a form of dimension reduction}. On the other hand, this reduction comes at the expense of relying on the assumption of linearity in (\ref{wthtage}). However, this is not so restrictive as it may seem, because: \begin{itemize} \item There are ways to assess the validity of the assumption. This is covered in almost any book on regression, such as mine (N. Matloff, \textit{Statistical Regression and Classification: from Linear Models to Machine Learning}, CRC, 2017). \item One can add polynomial terms, as seen in the next section. \item Assumptions tend to be less important in prediction contexts than in estimation. In the RS context, for instance, a rough model may be fine if we wish to take into account gender in predicting ratings, but might be insufficient if we want to estimate the actual magnitude of gender effect. \end{itemize} \subsection{The lm() Function in R} In R, the workhorse regression estimator is the \textbf{lm()} function. Let's apply this to the MovieLens data, predicting rating from age and gender. We'll define the latter as 1 for male, 0 for female. We find that our estimated regression function of rating on age and gender is \begin{equation} \textrm{estimated mean rating} = 3.3599 + 0.005311 ~ \textrm{age} - 0.0069 ~ \textrm{gender} \end{equation} (Note the word \textit{estimated}! These are not the true unknown population values, just estimates based on sample data.) Actually, this shows that age and gender are pretty weak predictors of movie rating, which you will recall is on a scale of 1 to 5. A 10-year difference in age raises the predicted rating only by about 0.05! The effect of gender is small too. And while it is interesting to see that older people tend to give slighly higher ratings, as do women, we must keep in mind that the magnitude of the effect here is small.\footnote{You may be familiar with the term \textit{statistically significant}. It is generally recognized today that this term can be quite misleading. This is beyond the scope of this book, but suffice it to say that although age and gender are statistically significant above (details available via adding the call \textbf{summary(lmout)} to the code below), their practical importance as predictors here is essentially nil. See R. Wasserstein and N. Lazar, The ASA's Statement on p-Values: Context, Process, and Purpose, \textit{The American Statistician}, June 2016.} Of course, the gender effect may be large in other RS datasets. Here is the annotated code: \begin{lstlisting} # read (user,item,rating,transID) data; name the columns ratings <- read.table('u.data') names(ratings) <- c('usernum','movienum','rating','transID') # read demographic data demog <- read.table('u.user',sep='|') names(demog) <- c('usernum','age','gender','occ','ZIP') # merge (database 'join' op) u.big <- merge(ratings,demog,by.x=1,by.y=1) u <- u.big[,c(1,2,3,5,6)] # fit linear model lmout <- lm(rating ~ age+gender,data=u) \end{lstlisting} Here's the output: \begin{lstlisting} > lmout Call: lm(formula = rating ~ age + gender, data = u) Coefficients: (Intercept) age genderM 3.359894 0.005311 -0.006904 \end{lstlisting} Let's take a closer look at that \textbf{genderM} coefficient.\footnote{The gender variable had been coded in the data as 'M' and 'F', and R chose the first one that showed up in the data, 'M', as its base.} Take for instance 28-year-old men and women; what are their mean ratings, according to this model? \begin{lstlisting} > lmout$coef %*% c(1,28,1) [,1] [1,] 3.50169 > lmout$coef %*% c(1,28,0) [,1] [1,] 3.508593 \end{lstlisting} (Note that the first '1' is needed to pick up the 3.359894.) So, on average, 28-year-old women give ratings 3.508593 - 3.50169 = 0.006903 higher than men of that age. And except for roundoff error, that is the -0.006904 value was see in the output above. \subsection{Details of Linear Regression Estimation} \label{lmdetails} In the weight-height-age example, say, we form \begin{equation} \label{rss} r = \sum_{i=1}^n [W_i - (b_0 + b_1 H_i + b_2 A_i)]^2 \end{equation} where $W_i$ is the weight of the i$^{th}$ person in our sample data and so on. This is the sum of squared prediction errors. We take derivatives with respect to the $b_k$ to minimize, then set $\widehat{\beta}_k$ to the minimizing $b_k$. Though R will do the minimizing for us, it is worth having an idea how it works, especially as more practice in following matrix-centric derivations. To get a glimpse of this, we look at a matrix formulation, as follows. Let $A$ denote the matrix of predictor values, with a 1s column tacked on at the left. In the above example, row 12, say, of $A$ would consist of a 1, followed by the height and age of the 12$^{th}$ person in our sample. Let $D$ denote the vector of weights, so that $D_{12}$ is the weight of the 12$^{th}$ person in our sample. Finally, let $b$ denote the vector of the $b_k$. Say we have data on 100 people. Then $A$ will have 100 rows, and $D$ will have length 100. Use the above as a concrete guide to your thinking, but keep in mind the general case: If we have $p$ predictors and $n$ data points, then $A$ and $D$ will have sizes $n \times (p+1)$ and $n$ Then \begin{equation} \label{} r = (D - A b)' (D - Ab) \end{equation} (Write it out to see this. Doing so will be crucial to understanding the material below and many points in the rest of the book.) Write the \textit{gradient} of $r$ with respect to $b$, \begin{equation} \frac{\partial r}{\partial b} = ( \frac{\partial r}{\partial b_0}, \frac{\partial r}{\partial b_1}, ..., \frac{\partial r}{\partial b_p} )' \end{equation} where $p+1$ is the number of predictor variables.\footnote{Note the representation here of a column vector as the transpose of a row vector. We will often do this, in order to save space on the page. And, any reference to a \textit{vector} will be to a column vector unless stated otherwise.} It can be shown that for a vector $u$, \begin{equation} \frac{\partial u'u}{\partial u} = 2u \end{equation} (analogous to the scalar relations $d (u^2) /du = 2u$; again, this is seen by writing the expressions out). Setting $u = D - Ab$ and applying the Chain Rule (adapted for gradients), we get \begin{equation} \frac{\partial r}{\partial b} = \frac{\partial r}{\partial u} \frac{\partial u}{\partial b} = 2(D - Ab) ~ \frac{\partial (D - Ab)}{\partial b}= 2 (-A') (D - Ab) \end{equation} Setting the gradient to 0 and solving for $b$, we have \begin{equation} \label{adaab} 0 = A'D - A'Ab \end{equation} so that the minimizing $b$, giving us $\widehat{\beta}$, is \begin{equation} \label{famouslm} b = (A'A)^{-1} A'D \end{equation} This famous formula is what \textbf{lm()} computes in finding the $\widehat{\beta}_k$. Note that we cannot simply multiply both sides of (\ref{adaab}) by $(A')^{-1}$, as $A'$ is nonsquare and thus noninvertible. Note too that in our age/gender MovieLens example above, we should not have variables for both male and female. If we did, we have \begin{equation} A = \left( \begin{array}{rrrr} 1 & A_1 & M_1 & F_1 \\ 1 & A_2 & M_2 & F_2 \\ ... \\ 1 & A_{100000} & M_{100000} & F_{100000} \\ \end{array} \right ) \end{equation} where $A_i$ is the age of the $i^{th}$ person in our data, and one of $M_{i}$ and $F_{i}$ is 1 and the other 0, according to the gender of this person. (Recall that there are 100000 data points in tis dataset.) The problem is this: The third and fourth columns of $A$ would then sum to a vector of all 1s, the same as in the first column. So the columns of $A$ will be linearly dependent, and the rank will be 3 instead of 4. The same will then be true for $A'A$, so that $(A'A)^{-1}$ will not exist in (\ref{famouslm}). In other words, not only would the F column be unnecessary, it would be prolematic. \subsubsection{Polynomial Terms} \label{poly} People tend to gain weight during middle age, but often they lose weight when they become elderly. So (\ref{wthtage}), which is linear in the age variable, may be rather unrealistic; we might believe a quadratic model for mean weight as a function of age is better: \begin{equation} \label{wthtage2} \textrm{mean weight} = \beta_0 + \beta_1 ~ \textrm{height} + \beta_2 ~ \textrm{age} + \beta_3 ~ \textrm{age}^2 \end{equation} A key point is that this is still a linear model! When we speak of a linear model --- the 'l' in ``lm()'' -- we mean linear in the $\beta_i$. If in (\ref{wthtage2}) we, say, multiply all the $\beta_i$ by 3, the entire sum grows by a factor of 3, hence the linearity in the $\beta_i$. Of course we may wish to add a quadratic term for height as well, and for that matter, a product term height $\times$ age. And since any model is merely an approximation, we might consider using higher and higher order polynomials. We do have to worry about overfitting though; see Section \ref{overfit}. We'll have a long example in Section \ref{prgengex}. \subsection{Categorical Variables (Predictor, Response)} A \textit{categorical} variable is one that codes categories. In our RS context, for instance, a user's postal code --- ZIP Code, in the US --- may be a helpful predictor of a rating. Yet it cannot be treated in \textbf{lm()}, say as a numeric variable. After all, the ZIP Code 90024, for example, is not ``twice as good'' as 45002; they are just ID codes. \subsubsection{Dummy Variables as Predictors} So, what do we do if we wish to use a categorical variable as a predictor? The answer is that we break the variable into a set of \textit{indicator variables}, colloquially known as \textit{dummy variables}. These have the values 1 and 0, with 1 \underline{indicating} the trait in question, 0 meaning not. Say for instance in RS we have the categorical variable State for users in a US state. We would define 50 dummy variables, one for each state. For instance, the one for California would have the value 1 if the user lives in California, 0 otherwise. Note carefully though that we would only use 49 of the dummies, not 50. We could for instance exclude Wyoming. Why? Because if the other 49 dummies are all 0, then we know this user must be in Wyoming. The Wyoming dummy would then be redundant. Not only do we want to avoid redundancy on dimension reduction grounds, but also that redundancy would result in the matrix $A$ in (\ref{famouslm}) being less than full rank, so $(A'A)^{-1}$ would not exist. \textit{Categorical variables in R:} In R, categorical variables are stored as objects of the class \textbf{'factor'}. The latter is a class designed specifically for representing categorical data. It consists of integer codes for the categories, with character-string names. The various categories are called \textit{levels}. For example, say we are dealing with eye color, and that the only colors are brown, blue and green, and have data on four people: \begin{lstlisting} > s <- c('brown','blue','green','blue') > class(s) [1] "character" > sf <- as.factor(s) > levels(sf) [1] "blue" "brown" "green" > sf[2] [1] blue Levels: blue brown green > as.numeric(sf) [1] 2 1 3 1 \end{lstlisting} In our MovieLens example above, let's take a look at the data frame \textbf{demog}: \begin{lstlisting} > for (i in 1:5) + print(class(demog[,i])) [1] "integer" [1] "integer" [1] "factor" [1] "factor" [1] "factor" \end{lstlisting} The last three columns are factors.\footnote{Even the first two could have been stored as factors, but were not coded as so.} Let's see how many occupations there are: \begin{lstlisting} > levels(demog$occ) [1] "administrator" "artist" "doctor" "educator" [5] "engineer" "entertainment" "executive" "healthcare" [9] "homemaker" "lawyer" "librarian" "marketing" [13] "none" "other" "programmer" "retired" [17] "salesman" "scientist" "student" "technician" [21] "writer" \end{lstlisting} In a regression application, we'd form 21 dummies, but use only 20 of them (any 20). However, the designers of R (and its predecessor S), in their wisdom, set things up to save us some time and trouble. We can just specify the factors we wish to use, and R will form the dummies for us, being careful to drop one of them. This is what happened, for instance, in our example above in which we regressed rating against age and gender, with output \begin{lstlisting} Coefficients: (Intercept) age genderM 3.359894 0.005311 -0.006904 \end{lstlisting} R noticed that gender was an R factor, with levels M and F. It created dummies for M and F, but just retained the former, as the first case in the data had gender as M. So, the estimated coefficient -0.006904 meant that the ``maleness'' impact on mean rating has that value. Men give slightly lower ratings than women do, for fixed age. In this manner, the estimated regression coefficient of a dummy variable is the effect, all other predictors fixed, of this trait \textit{relative to the excluded trai}, say femaleness above. \subsubsection{Interaction Terms} Say we are in some RS context in which age and gender are substantial factors in predicting rating. Suppose also that we suspect men become more liberal raters as they age while women become more reserved in their ratings. Then a model like this might work well: \begin{equation} \label{mfinteraction} \textrm{mean rating} = \beta_0 + \beta_1 ~ \textrm{age} + \beta_2 ~ \textrm{male} + \beta_3 ~ \textrm{age} \times \textrm{male} \end{equation} where \textit{male} is a dummy variable. To see why this might be appropriate, consider what the above equation reduces to for men and women: \textit{men:} \begin{equation} \textrm{mean rating} = (\beta_0+\beta_2) + (\beta_1+\beta_3) ~ \textrm{age} \end{equation} \textit{women:}M \begin{equation} \textrm{mean rating} = \beta_0 + \beta_1 ~ \textrm{age} \end{equation} So the male and female lines have different slopes (and different intercepts), allowing for the differential age effect we surmise. Of course, once we compute the $\widehat{\beta}_i$ from the data, it may well turn out that our differential aging trends may not be confirmed.\footnote{One must take sampling variability into account, say by forming confidence intervals for the $\beta_i$. As noted earlier, do not use significance testing for this. At any rate, these aspects are beyond the scope of this book.} The term $\textrm{age} \times \textrm{male}$ is called an \textit{interaction term}. Note that interaction terms can be formed from any predictor, not just dummy variables. Also, one can form triple products for three-way interactions and so on, though this could greatly increase the complexity of the model and thus risk overfitting. \subsubsection{Dummy Variables as Response Variables} In many cases, the response variable may be categorical. In the RS context, for instance, a rating may simply be binary, i.e.\ like/dislike. Or even click/not click --- does a user click on a Web page location? Let's use this as our example. We are generally interested in the probability of a click. That actually fits a regression context, as follows. Code a click as 1 and nonclick as 0. Since the expected value of a variable of this type is the probability of a 1, and since a regression function by definition is an expected value, taking Click as our response variables does involve a regression function. So, if our predictors were age and gender, say, we might entertain formulating our regression model as \begin{equation} \label{clickagegen} \textrm{probability of click} = \beta_0 + \beta_1 ~ \textrm{age} + \beta_2 ~ \textrm{gender} \end{equation} One problem, though, is that a probability should be in [0,1] yet the right-hand side of (\ref{clickagegen}) can conceivably be anywhere in $(-\infty,\infty)$. For this and other reasons the usual parametric model for a binary response $Y$ is the \textit{logistic}: For $p$ predictors $X_i$, our model is \begin{equation} P(Y = 1 ~|~ X_1=t_1,...,X_p=t_p) = \frac{1}{1+\exp{-(\beta_0+\beta_1 t_1+...+\beta_p t_p})} \end{equation} This is called a \textit{generalized linear model}, as it has the linear form $\beta_0+\beta_1 t_1+...+\beta_p t_p$ embedded inside another function, in this case the logistic function $g(s) = 1/(1+e^{-s})$. Note that the latter function, often called \textit{logit} for short, has values only in [0,1], as desired, and is increasing in $s$, thus retaining the monotonic notion of linear models.\footnote{These properties form the intuitive motivation for using logit models. Another motivation is this: Let $X$ denote the vector of predictor variables, and let $Y$ be the response variable, with the two classes 0 and 1. If within each class, $X$ has a multivariate normal distribution, with the same covariance matrix in each class.} The $\beta_i$ are estimated by an R function \textbf{glm()}, similar to \textbf{lm()}.\footnote{The class of the return value is \textbf{'glm'}, which is a subclass of \textbf{'lm'.} } Let's model a user giving a movie a rating of 4 or higher: \begin{lstlisting} > r45 <- as.integer(u$rating >= 4) # a binary value, 1 or 0 > u$r45 <- r45 > glmout <- glm(r45 ~ age+gender,data=u,family=binomial) > glmout Call: glm(formula = r45 ~ age + gender, data = u) Coefficients: (Intercept) age genderM -0.002510 0.006886 -0.011189 ... \end{lstlisting} The argument \textbf{family = binomial} tells R that we want the logistic model, not some other generalized linear model, such a model known as \textit{Poisson regression}.\footnote{By the way, the argument \textbf{family} must be an object of class \textbf{'function'}. Inside \textbf{glm()}, there will be a call \textbf{family()}. R has a built-in function \textbf{binomial()}, which is called here.} \subsection{R's predict(), a Generic Function} A key aspect to R's object orientation is \textit{generic} functions. Take \textbf{plot()}, for instance. Its action will depend on the class of object it is applied to. If we call the function on a vector, we get a histogram. But if we call it on a two-column matrix, we get a scatter diagram. What happens is that when \textbf{plot()} is called, R will check what class of object the caller supplied as an argument. If the object is of class \textbf{"x"}, then the original call will be \textit{dispatched} to \textbf{plot.x()}, a plotting function tailored to that class. (Of course, that means one needs to have been written and available.) R's \textbf{predict()} is another example of a generic function, used to predict new cases. In the MovieLens example above, say we want to predict the rating given by a 30-year-old man. We could simply plug 30 and 1 into the estimated regression function, say using \textbf{coef()} to get the $\widehat{\beta}_i$: \begin{lstlisting} > coef(lmout) (Intercept) age genderM 3.359894442 0.005310673 -0.006903502 > coef(lmout) %*% c(1,30,1) # linear algebra-style matrix multiply [,1] [1,] 3.512311 \end{lstlisting} Alternatively (and in many settings, more conveniently): \begin{lstlisting} > newdata <- data.frame(age=30,gender='M') > predict(lmout,newdata) 1 3.512311 \end{lstlisting} Recall that we had assigned the output of \textbf{lm()} to \textbf{lmout}, which will have class \textbf{'lm'}. So, the call to \textbf{predict()} above was dispatched to \textbf{predict.lm()}. What about \textbf{glm()}? There is a function \textbf{predict.glm()}, which normally should be called with the argument \textbf{type = 'response'}. The latter means we want the return values to be the estimated values of the regression function, i.e.\ the conditional probabilities of response 1, given the values of the predictors. \subsection{More Than Two Levels in Categorical Response} What if our response variable is categorical but with more than two levels? In the click/nonclick setting, suppose the user has a choice of five things to click, and must choose one. Then the response is categorical with five levels. There are two major approaches. To explain, we'll use the following very simple example. Say there are dogs, cats and foxes on a field, and they sometimes step on a sensor, so we know their weights but do not see them. Say we have data on 10000 data points, in which we do know the species. Our data frame, \textbf{df}, has 10000 rows and 4 columns. In the columns, say the names are 'Weight', 'Dog', 'Cat' and 'Fox', with the last three being dummies. Say we have 5000 dogs, 2000 cats and 3000 foxes. Then for instance 2000 of the rows in \textbf{df} would be cats. \textit{One-vs.All (OVA) Method} One would run run three logistic models: \begin{lstlisting} gdog <- glm(Dog ~ .,data=df[,1:2]) # dog vs. all else gcat <- glm(Cat ~ .,data=df[,c(1,3)]) # cat vs. all else gfox <- glm(Fox ~ .,data=df[,c(1,4]) # fox vs. all else \end{lstlisting} Then for each new animal we encounter of unknown species, we call \textbf{predict()} three times, yielding three estimated conditional probabilities. If the one for cat, say, is largest, we guess Cat. \textit{All vs. All (AVA) Method} Here again we'd run multiple logit models, in pairs as follows: \begin{lstlisting} gdogcat <- glm(Dog ~ .,data=df[df$dog+df$cat==1,1:2]) # dog vs. cat gdogfox <- glm(Dog ~ .,data=df[df$dog+df$fox==1,1:2]) # dog vs. fox gcatfox <- glm(Cat ~ .,data=df[df$cat+df$fox==1,1:3]) # cat vs. fox \end{lstlisting} Then for each new animal we encounter of unknown species, we call \textbf{predict()} three times, again yielding three estimated conditional probabilities. Say in the first one, Cat ``wins,'' i.e.\ the conditional probability is less than 0.5. Say Dog wins in the second, and Cat wins in the third. Since Cat had the most wins, we predict Cat. \textit{Comparison} At first, OVA seems much better than AVA. If we have $m$ levels, that means running $C(m,2) = O(m^2)$ parwise logit models, rather than $m$ for OVA. However, that is somewhat compensated by the fact that each pairwise model will be based on less data, and some analysts contend that AVA can have better accuracy. It remains a bit of a controversy. \section{Bias, Variance, Overfitting and p-Hacking} \label{overfit} By far the most vexing issue in statistics and machine learning is that of \textit{overfitting}. \subsection{What Is Overfitting?} \label{norelation} Suppose we have just one predictor, and $n$ data points. If we fit a polynomial model of degree $n-1$, the resulting curve will pass through all $n$ points, a ``perfect'' fit. For instance: \begin{lstlisting} > x <- rnorm(6) > y <- rnorm(6) # unrelated to x! > df <- data.frame(x,y) > df$x2 <- x^2 > df$x3 <- x^3 > df$x4 <- x^4 > df$x5 <- x^5 > df x y x2 x3 1 -1.1855131 0.2881291 1.40544120 -1.666168894 2 -1.7838769 -2.0741740 3.18221664 -5.676682627 3 -0.7124510 -0.4253678 0.50758640 -0.361630431 4 0.1676111 -0.1949265 0.02809348 0.004708779 5 1.2462926 -0.7348481 1.55324535 1.935798245 6 0.3741604 1.9521667 0.13999601 0.052380963 x4 x5 1 1.975265e+00 -2.341702414 2 1.012650e+01 -18.064433938 3 2.576440e-01 -0.183558689 4 7.892437e-04 0.000132286 5 2.412571e+00 3.006769615 6 1.959888e-02 0.007333126 > lmo <- lm(y ~ .,data=df) > lmo Call: lm(formula = y ~ ., data = df) Coefficients: (Intercept) x x2 x3 -1.3127 4.7632 11.4809 0.5781 x4 x5 -6.9685 -2.4938 > lmo$fitted.values 1 2 3 4 5 0.2881291 -2.0741740 -0.4253678 -0.1949265 -0.7348481 6 1.9521667 > y [1] 0.2881291 -2.0741740 -0.4253678 -0.1949265 -0.7348481 [6] 1.9521667 \end{lstlisting} Yes, we ``predicted'' \textbf{y} perfectly, \textbf{even though there was no relation between the response and predictor variables).} Clearly that ``perfect fit'' is illusory, ``noise fitting.'' Our ability to predict future cases would not be good. This is \textit{overfitting}. Let's take a closer look, in an RS context. Say we believe (\ref{mfinteraction}) is a good model for the setting described in that section, i.e.\ men becoming more liberal raters as they age but women becoming more conservative. If we omit the interaction term, than we will underpredict older men and overpredict older women. This biases our ratings. On the other hand, we need to worry about sampling variance. Consider the case of opinion polls during an election campaign, in which the goal is to estimate $p$, the proportion of voters who will vote for Candidate Jones. If we use too small a sample size, say 50, our results will probably be inaccurate. This is due to sampling instability: Two pollsters, each randomly sampling 50 people, will sample different sets of people, thus each having different values of $\widehat{p}$, their sample estimates of $p$. For a sample of size 50, it is likely that their two values of $\widehat{p}$ will be substantially different from each other, whereas if the sample size were 5000, the two estimates would likely be close to each other. In other words, the \underline{variance} of $\widehat{p}$ is too high if the sample size is just 50.\footnote{The repeatable experiment here is randomly choosing 50 people. Each time we perform this experiment, we get a different set of 50 people, thus a different value of $\widehat{p}$. The latter is a random variable, and thus has a variance.} In a parametric regression setting, increasing the number of terms roughly means that the sampling variance of the $\widehat{\beta}_i$ will increase. So we have the famous \textit{bias/variance tradeoff}: As we use more and more terms in our regression model (predictors, polynomials, interaction terms), the bias decreases but the variance increases. This ``tug of war'' between these decreasing and increasing quantities typically yields a U-shaped curve: As we increase the number of terms from 1, mean absolute prediction error will at first decrease but eventually will increase. Once we get to the point at which it increases, we are \textit{overfitting}. This is particularly a problem when one has many dummy variables. For instance, there are more than 42,000 ZIP Codes in the US; to have a dummy for each would almost certainly be overfitting. \subsection{Can Anything Be Done about It?} So, where is the ``happy medium,'' the model that is rich enough to capture most of the dynamics of the variables at hand, but simple enough to avoid variance issues? Unfortunately, \textbf{there is no good answer to this question.} One quick rule of thumb is that one should have $p < \sqrt{n}$, where $p$ is the number of predictors, including polynomial and interaction terms (not to be confused with the quantity of the same name in our polling example above), and $n$ is the number of cases in our sample. But this is certainly not a firm rule by any means. One may be able to get good prediction with a considerably larger value of $p$, especially if \textit{regularization} methods are used (Chapter \ref{chap:svd}). From the polynomial-ftting example in Section \ref{norelation}, we see the following key point: \begin{quote} An assessment of predictive ability, based on predicting the same data on which our model is fit, tends to be overly optimistic and may be meaningless or close to it. \end{quote} This motivates the most common approach to dealing with the bias/variance tradeoff, \textit{cross validation}. In the simplest version, one randomly splits the data into a \textit{training set} and a \textit{test set}.\footnote{The latter is also called a \textit{holdout set} or a \textit{validation set}.} We fit the model to the training set and then, pretending we don't know the ``Y'' (i.e. response) values in the test set, predict those values from our fitted model and the ``X'' values (i.e.\ the predictors) in the test set. We then ``unpretend,'' and check how well those predictions worked. The test set is ``fresh, new'' data, since we called \textbf{lm()} or whatever only on the training set. Thus we are avoiding the ``noise fitting'' problem. We can try several candidate models, then choose the one that best predicts the test data. (Note carefully that after fitting the model via cross-validation, we then use the full data for later prediction. Splitting the data for cross-validation was just a temporary device for model selection.) Cross-validation is essentially the standard for model selection, and it works well if we only try a few models. Problems can occur if we try many models, as seen in the next section. \subsection{The Problem of P-hacking} The (rather recent) term \textit{p-hacking} refers to the following abuse of statistics.\footnote{The term \textit{abuse} here will not necessarily connote intent. It may occur out of ignorance of the problem.} Say we have 250 pennies, and we wish to determine whether any are unbalanced, i.e.\ have probability of heads different from 0.5. We do so by tossing each coin 100 times. If we get fewer than 40 heads or more than 60, we decide this coin is unbalanced.\footnote{For those who know statistics: This gives us a Type I error rate of about 0.05, the standard used by most people.} The problem is that, even if all the coins are perfectly balanced, we eventually will have one that has fewer than 40 or greater than 60 heads, just by accident. \textbf{We will then \underline{falsely} declare this coin to be unbalanced.} Or, to give a somewhat frivolous example that still will make the point, say we are investigating whether there is any genetic component to a person's sense of humor. Is there a Humor gene? There are many, many genes to consider. Testing each one for relation to sense of humor is like checking each penny for being unbalanced: Even if there is no Humor gene, then eventually, just by accident, we'll stumble upon one that seems to be related to humor.\footnote{For those with background in statistics, the reason this is called ``p-hacking'' is that the researcher may form a significance test for each gene, computing a p-value for each test. Since under the null hypothesis we have a 5\% chance of getting a ``significant'' p-value for any given gene, the probability of having at least one significant result out of the thousands of tests is quite high, even if the null hypothesis is true in all cases. There are techniques called \textit{multiple inference} or \textit{multiple comparison} methods, to avoid p-hacking in performing statistical inference. See for example \textit{Multiple Comparisons: Theory and Methods}, Jason Hsu, 1996, CRC.} Though the above is not about prediction, it has big implications for the prediction realm. In ML there are various datasets on which analysts engage in contests, vying for the honor of developing the model with the highest prediction accuracy, say for classification of images. If there is a large number of analysts competing for the prize, then even if all the analysts have models of equal accuracy, it is likely that one is substantially higher than the others, just due to an accident of sampling variation. This is true in spite of the fact that they all are using the same sample; it may be that the ``winning'' analyst' model happens to do especially well in the given data, and may not be so good on another sample from the same population. So, when some researcher sets a new record on a famous ML dataset, it may be that the research really has found a better prediction model --- or it may be that it merely looks better, due to p-hacking. The same is true for your own analyses. If you try a large number of models, the ``winning'' one may actually not be better than all the others. \section{Extended Example} \label{prgengex} Let's illustrate this on the dataset \textbf{prgeng}, assembled from the 2000 US census. It consists of wage and other information on programmers and engineers in Silicon Valley. This dataset is included in the R \textbf{polyreg} package, which fits polynomial models as we saw in Section \ref{poly} above.\footnote{Available from \url{github.com/matloff}.} \begin{lstlisting} > getPE() # produces data frame pe > pe1 <- pe[,c(1,2,4,6,7,12:16,3)] # choose some predictors > head(pe1) age sex wkswrkd ms phd occ1 occ2 occ3 occ4 occ5 1 50.30082 0 52 0 0 0 0 1 0 0 2 41.10139 1 20 0 0 0 1 0 0 0 3 24.67374 0 52 0 0 0 0 1 0 0 4 50.19951 1 52 0 0 1 0 0 0 0 5 51.18112 0 1 0 0 1 0 0 0 0 6 57.70413 1 0 0 0 1 0 0 0 0 wageinc 1 75000 2 12300 3 15400 4 0 5 160 6 0 \end{lstlisting} By the way, note the dummy variables. We have just two levels for education, so anyone with just a bachelor's degree or less, or a professional degree, with be ``other,'' coded by \textbf{ms} and \textbf{phd} both having the values 0. Similarly, there are actually six occupations, hence five dummies; the sixth occupation is coded by having 0s for the five occupation dummies. Well, then, let's see how well we can predict wage income. First, split the data: \begin{lstlisting} > set.seed(9999) > testidxs <- sample(1:nrow(pe1),1000) > testset <- pe1[testidxs,] > trainset <- pe1[-testidxs,] \end{lstlisting} Then fit the model and predict: \begin{lstlisting} > lmout <- lm(wageinc ~ .,data=pe1) # predict wage from all others > predvals <- predict(lmout,testset[,-11]) > mean(abs(predvals - testset[,11])) [1] 26494.62 \end{lstlisting} An average, our prediction is off by more than \$26,000, not so good, but not the main point here. Let's try a quadratic model. Again, we could form the squared terms ourselves, but \textbf{polyreg} makes it convenient. \begin{lstlisting} > pfout <- polyFit(pe1,deg=2) > mean(abs(predict(pfout,testset[,-11]) - testset[,11])) [1] 25885.43 \end{lstlisting} Ah, some improvement, with a second-degree model. Note that this includes the squares of all predictors, and their products, e.g.\ age times weeks worked. Altogether there are now $p = 46$ predictors, up from our original 10: \begin{lstlisting} > dim(pfout$poly.xy) [1] 20090 47 \end{lstlisting} How about a cubic model? \begin{lstlisting} > pfout <- polyFit(pe1,deg=3) > mean(abs(predict(pfout,testset[,-11]) - testset[,11])) [1] 25410.9 \end{lstlisting} Even better, though it may be largely a difference due to sampling variation. We are now up to $p = 118$. \begin{lstlisting} > pfout <- polyFit(pe1,deg=4) getPoly time: 1.12 0.056 1.177 0 0 lm() time: 0.88 0.024 0.906 0 0 > mean(abs(predict(pfout,testset[,-11]) - testset[,11])) [1] 25380.65 Warning message: In predict.lm(object$fit, plm.newdata) : prediction from a rank-deficient fit may be misleading \end{lstlisting} About the same, but with an ominous warning. R found that the matrix $A'A$ in (\ref{famouslm}) was close to nonfull-rank, thus nearly singular (noninvertible). Now $p = 226$. Remarkably, this improvement continued: \bigskip \begin{tabular}{|r|r|} \hline deg & MAPE \\ \hline 6 & 25245.83 \\ \hline 10 & 24679.86 \\ \hline 15 & 24123.59 \\ \hline \end{tabular} And at degree 15, $p$ was a whopping 3791. By the way, in that last computation, it took more than 2 minutes to simply generate the polynomials --- those 3791 columns --- and almost 20 minutes to process the call to \textbf{lm()}. Yet, even with this large model, we have not yet reached the point of overfitting. Eventually, though, MAPE will begin to rise.