\chapter{Some Infrastructure: Linear Algebra} \label{chap:linalg} There are some issues that will come up frequently. We'll first cover them briefly here, more later as the need arises. This chapter will review linear algebra, while the following one will review/extend the reader's knowledge of probability and statistics. RS methods, as with other machine learning (ML) techniques, often make use of linear algebra, well beyond mere matrix multiplication. \section{Matrix Rank and Vector Linear Independence} Consider the matrix \begin{equation} \label{rankex1} M = \left ( \begin{array}{rrrr} 1 & 5 & 1 & -2 \\ 8 & 3 & 2 & 8 \\ 9 & 8 & 3 & 6 \end{array} \right ) \end{equation} Note that the third row is the sum of the first two. In many contexts, this would imply that there are really only two ``independent'' rows in $M$ in some sense related to the application. Denote the rows of $M$ by $r_i, ~ i = 1,2,3$. Recall that we say they are \textit{linearly independent} if it is not possible to find scalars $a_i$, at least one of them nonzero, such that the \textit{linear combination} $a_1 r_1 + a_2 r_2 + a_3 r_3$ is equal to 0. In this case $a_1 = a_2 = 1$ and $a_3 = -1$ gives us 0, so the rows of $M$ are linearly dependent. Recall that the \textit{rank} of a matrix is its maximal number of linearly independent rows and columns. (It can be shown that the row rank and column rank are the same.) The rank of $M$ above is 2. Recall too the notion of the \textit{basis} of a vector space $\mathcal{V}$. It is a linearly independent set of vectors whose linear combinations collectively form all of $\mathcal{V}$. Here $r_1$ and $r_2$ form a basis for the \textit{row space} of $M$. Alternatively, $r_1$ and $r_3$ also form a basis, as do $r_2$ and $r_3$. The rank of an $r \times s$ matrix is thus at most $\min(r,s)$. In the case of equality, we say the matrix has \textit{full rank}. A ratings matrix, such as $A$ in Section \ref{mf}, should be of full rank, since there presumably are no exact dependencies among users or items. \section{Partitioned Matrices} \label{partmat} It is often helpful to partition a matrix into \textit{blocks} (often called {\it tiles} in the parallel computation community). \subsection{How It Works} Consider matrices $A$, $B$ and $C$, \begin{equation} A = \left ( \begin{array}{ccc} 1 & 5 & 12 \\ 0 & 3 & 6 \\ 4 & 8 & 2 \end{array} \right ) \end{equation} and \begin{equation} B = \left ( \begin{array}{ccc} 0 & 2 & 5 \\ 0 & 9 & 10 \\ 1 & 1 & 2 \end{array} \right ), \end{equation} so that \begin{equation} C = AB = \left ( \begin{array}{ccc} 12 & 59 & 79 \\ 6 & 33 & 42 \\ 2 & 82 & 104 \end{array} \right ) . \end{equation} We could partition A as, say, \begin{equation} A = \left ( \begin{array}{cc} A_{11} & A_{12} \\ A_{21} & A_{22} \end{array} \right ) , \end{equation} where \begin{equation} A_{11} = \left ( \begin{array}{cc} 1 & 5 \\ 0 & 3 \end{array} \right ) , \end{equation} \begin{equation} A_{12} = \left ( \begin{array}{cc} 12 \\ 6 \end{array} \right ), \end{equation} \begin{equation} A_{21} = \left ( \begin{array}{cc} 4 & 8 \end{array} \right ) \end{equation} and \begin{equation} A_{22} = \left ( \begin{array}{c} 2 \end{array} \right ). \end{equation} Similarly we would partition B and C into blocks of a compatible size to A, \begin{equation} B = \left ( \begin{array}{cc} B_{11} & B_{12} \\ B_{21} & B_{22} \end{array} \right ) \end{equation} \begin{equation} C = \left ( \begin{array}{cc} C_{11} & C_{12} \\ C_{21} & C_{22} \end{array} \right ) , \end{equation} so that for example \begin{equation} B_{21} = \left ( \begin{array}{cc} 1 & 1 \end{array} \right ) . \end{equation} The key point is that multiplication still works if we pretend that those submatrices are numbers! For example, pretending like that would give the relation \begin{equation} C_{11} = A_{11} B_{11} + A_{12} B_{21}, \end{equation} which the reader should verify really is correct as matrices, i.e. the computation on the right side really does yield a matrix equal to $C_{11}$. \subsection{Important Special Case: Matrix Times Vector} Consider the product of a matrix and a vector, $Ax$. This product is a linear combination of the columns of $A$. To see this, write \begin{equation} \label{amatrix} A = (A_1, A_2, A_3) \end{equation} with $A_i$ being the $i^{th}$ column, and \begin{equation} x = \left ( \begin{array}{c} x_1 \\ x_2 \\ x_3 \end{array} \right ) \end{equation} The point is that (\ref{amatrix}) looks like a row vector --- it isn't, but we pretend it is --- so that $Ax$ looks like the ``dot product of one vector with another. That would give us \begin{equation} \label{ax} Ax = x_1 A_1 + x_2 A_2 + x_3 A_3 \end{equation} As noted, we then ``unpretend,'' and find that (\ref{ax}) says that \begin{quote} $Ax$ is a linear combination of the columns of $A$. The coefficients in that linear combination are the elements of $x$. \end{quote} \subsection{Example: Matrix Factorization} \label{mfact} Recall the relation \begin{equation} A \approx WH \end{equation} in Section \ref{mf}, where $A$ is $r \times s$, $W$ is $r \times m$ and $H$ is $m \times s$. Partition the first and third matrices into rows, i.e.\ write \begin{equation} A = \left ( \begin{array}{c} a_1 \\ ... \\ a_r \end{array} \right ), \end{equation} and \begin{equation} H = \left ( \begin{array}{c} h_1 \\ ... \\ h_m \end{array} \right ), \end{equation} so that for instance $a_1$ and $h_1$ are the first row in $A$ and $H$, respectively. Keep $W$ unpartitiond: \begin{equation} W = \left ( \begin{array}{ccc} w_{11} & ... & w_{1m}\\ ... & ... & ... \\ w_{r1} & ... & w_{rm} \end{array} \right ), \end{equation} Using the partitioning idea, write $WH$ as a ``matrix-vector product'': \begin{equation} WH = \left ( \begin{array}{ccc} w_{11} & ... & w_{1m}\\ ... & ... & ... \\ w_{r1} & ... & w_{rm} \end{array} \right ) \left ( \begin{array}{cc} h_1 \\ ... \\ h_m \end{array} \right ) = \left ( \begin{array}{cc} w_{11}h_1 + ... + w_{1m} h_m \\ ... \\ w_{r1}h_1 + ... + w_{rm} h_m \\ \end{array} \right ) \end{equation} Look at that! What it says is row $i$ of $WH$, and thus approximately row $i$ of $A$, is a linear combination of the rows of $H$. And with a different partitioning, we'd find that each column of $WH$ is a linear combination of the columns of $H$. We'll see in Chapter \ref{chap:svd} that this has big implications for the matrix factorization method of RS, a topic we lay the groundwork for next. \section{The Notion of Approximate Rank: Principal Components Analysis} Suppose the matrix in (\ref{rankex1}) had been \begin{equation} \label{rankex2} M = \left ( \begin{array}{rrrr} 1 & 5 & 1 & -2\\ 8.02 & 2.99 & 2 & 8.2\\ 9 & 8 & 3 & 6 \end{array} \right ) \end{equation} Intuitively, we still might say that the rank of $M$ is ``approximately'' 2. Or better yet, row 3 still seems redundant, Let's formalize that, leading to one of the most common techniques in statistics/machine learning. By the way, this technique will also later provide a way to find $W$ and $H$ in (\ref{awh}). (Warning: This section will be somewhat long, but quite central to RS/ML.) \subsection{Dimension Reduction} One of the major themes in computer science is \textit{scale}, as in the common question, ``Does it scale?'' The concern is, does an algorithm, method or whatever work well in large-scale systems? In the RS context, just think of, say, Amazon. The business has millions of users and millions of items. In other words, the ratings matrix has millions of rows and millions of columns, and even one million rows and columns would mean a total number of $(10^6)^2 = 10^{12}$ entries, about 8 terabytes of data. This is a core point in statistics/machine learning, the notion of \textit{dimension reduction}. In complex applications, there is a pressing need to reduce the number of variables down to a manageable number --- manageable not only in terms of computational time and space, but also the statistical problem of \textit{overfitting} (Chapter \ref{chap:infra2}). So we need methods to eliminate redundant or near-redundant data, such as row 3 in (\ref{rankex2}). \subsection{Exploiting Correlations} \label{explorecorr} Statistically, the issue is one of correlation. In (\ref{rankex2}), the third row is highly correlated with (the sum of) the first two rows. To explore the correlation idea further, here are two graphs of bivariate normal densities: \begin{minipage}[b]{0.65\linewidth} \mbox{\includegraphics[width=3.25in]{Images/Rho2.pdf}} \end{minipage} \hspace{0.0in} \begin{minipage}[b]{0.65\linewidth} \mbox{\includegraphics[width=3.25in]{Images/Rho8.pdf}} \end{minipage} Let's call the two variables $X_1$ and $X_2$, say human height and weight, with the corresponding axes in the graphs to be referred to as $t_1$ and $t_2$. The first graph was generated with a correlation of 0.2 between the two variables, while in the second one, the correlation is 0.8. Not surprisingly due to the high correlation in the second graph, the ``two-dimensional bell'' is concentrated around a straight line, specifically the line $t_2 = - t_1$. In other words, there is high probability that $X_2 \approx -X_1$, so that $X_2$ is largely redundant. So: \begin{quote} To a large extent, there is only one variable here, $X_1$ (or other choices, e.g.\ $X_2$), not two. \end{quote} In the case of correlation 0.2, there really are two separate variables. The probability that $X_2 \approx -X_1$ is lower here. Note one more time, though, the approximate nature of the approach we are developing. There really \textit{are} two variables even in that correlation 0.8 example. By using only one of them, \textbf{we are relinquishing some information. But with the need to avoid overfitting, use of the approximation may be a net win for us.} Well then, how can we determine a set of near-redundant variables, so that we can consider omitting them from our analysis? Let's look at those graphs a little more closely. Any \textit{level set} in the above graphs, i.e.\ a curve one obtains by slicing the bells parallel to the $(t_1,t_2)$ plane can be shown to be an ellipse. As noted, the major axis of the ellipse will be the line $t_1 + t_2 = 0$. The minor axis will be the line perpendicular to that, $t_1 - t_2 = 0$. In turn, that means that standard probability methods can then be used to show that the variables \begin{equation} Y_1 = X_1 + X_2 \end{equation} and \begin{equation} Y_2 = X_1 - X_2 \end{equation} have 0 correlation. Then we have a good case for using only $Y_1$ in our data analysis, instead of using $X_1$ and $X_2$. But why not use just $X_1$? As usual in statistics/ML, things get more complicated in higher dimensions. In choosing variables to retain in our analysis, it makes sense to require that they be uncorrelated, as $Y_1$ and $Y_2$ are above; if not, intuitively there is some redundancy among them, which of course is what we are hoping to avoid. Remember, we want to reduce the number of variables we'll work with, and redundancy would say that we might be able to reduce further. With that in mind, now suppose we have $p$ variables, $X_1, X_2,...,X_p$, not just two. If our data is on people, these variables may be height, weight, age, gender and so on. We can no longer visualize in higher dimensions, but one can show that the level sets will be $p$-dimensional ellipsoids. These now have $p$ axes rather than just two, and we can define $p$ new variables, $Y_1,Y_2,...,Y_p$ from the $X_i$, such that: \begin{itemize} \item [(a)] The $Y_i$ are uncorrelated. \item [(b)] They are ordered in terms of variance: \begin{equation} Var(Y_1) > Var(Y_2) > ... > Var(Y_p) \end{equation} \end{itemize} Now we have a promising solution to our dimension reduction problem. In [(b)] above, we can choose to use just the first few of the $Y_i$, omitting the ones with small variance since they are essentially constants, uninformative. And again, since the $Y_i$ will be uncorrelated, we are eliminating a source of possible redundancy among them. PCA won't be a perfect solution --- there is no such thing --- as might be the case if the relations between variables is nonmonotonic. A common example is age, with mean income given age tending to be a quadratic (or higher degree) polynomial relation. But PCA is a very common ``go to'' method for dimension reduction, and may work well even in (mildly) nonmonotonic settings. Now, how do we find these $Y_i$? \subsection{Eigenanalysis} Say I have a sample of $n$ observations on two variables, $x = (x_1,...,x_n)$ and $y=(y_1,...,y_n$, say height and weight on $n = 100$ people. Then, formally, the \textit{correlation} between the variables is \begin{equation} \label{corrdef} \frac {\frac{1}{n} \sum_{i=1}^n (x_i-\overline{x}) (y_i-\overline{y})} {s.d.(x) ~ s.d.(y)} \end{equation} where the denominator is the product of the sample standard deviations of the two variables, and $\overline{x})$ and $\overline{y})$ are the sample means. The correlation is a number between -1 and 1. The correlation matrix $C$ --- i.e. the set of all $\textrm{corr}(X_i,X_j)$ --- of a set of $p$ variables is $p \times p$, i.e.\ square. Moreover, since $\textrm{corr}(X_i,X_j) = \textrm{corr}(X_j,X_i)$, $C$ is \textit{symmetric}: \begin{equation} C' = C \end{equation} where $'$ denotes matrix transpose. Recall that for any square matrix $L$, if there is a scalar $\lambda$ and a nonzero vector $x$ such that \begin{equation} Lx = \lambda x \end{equation} then we say that $x$ is an \textit{eigenvector} of $L$, with \textit{eigenvalue} $\lambda$. (Note that the above implies that $x$ is a column vector, $p \times 1$, a convention we will use throughout the book.) It can be shown that any symmetric matrix has real (not complex) eigenvalues, and that the corresponding eigenvectors $U_1,...,U_p$ are \textit{orthogonal}, \begin{equation} U_i' U_j = 0, ~ i \neq j \end{equation} We always take the $U_i$ to have length 1: Just divide the vector by its length, so it now has length 1, and is still an eigenvector. \subsection{PCA} Typically we have many cases in our data, say $n$, arranged in an $n \times p$ matrix $Q$, with row $i$ representing case $i$ and column $j$ representing the $j^{th}$ variable. Say our data is about people, 1000 of them, and we have data on height, weight, age, gender, years of schooling and income. Then $n = 1000$, $p = 6$. So, finally, here is PCA: \begin{enumerate} \item Find the correlation matrix (or covariance matrix, a similar notion) from the data in $Q$. Note again that since there are $p$ variables, the correlation matrix will be $p \times p$. \item Compute its eigenvalues and eigenvectors. It can be shown that the eigenvalues are the variances of our new variables. \item After ordering the eigenvalues from largest to smallest, let $\lambda_i$ be the $i^{th}$ largest, and let $U_i$ be the corresponding eigenvector, scaled to have length 1. \item Let $U$ be the matrix whose $i^{th}$ column is $U_i$. Its size will be $p \times p$. \item Choose the first few eigenvalues, say $s$ of them, using some criterion (see below). Denote the matrix of the first $s$ columns of $U$ by $U^{(s)}$. \item Form a new data matrix, \begin{equation} \label{rqu} R = Q U^{(s)} \end{equation} $R$ will be of size $n \times s$. So, we've replaced our original $p$ variables, the $p$ columns of $Q$, by $s$ new variables, the columns of $R$. Column $j$ of $R$ is called the $j^{th}$ principal component of the original data. As mentioned, it can be shown that the variance of the $j^{th}$ principal component is $\lambda_j$. The sum of all $p$ eigenvalues is the same as the sum of the variances of the original variables, an important point. \end{enumerate} From this point onward, any data analysis we do will be with $R$, not $Q$. In $R$, row $i$ is still data on the $i^{th}$ case, e.g. the $i^{th}$ person, but now with $s$ new variables in place of the original $p$. Since typically $s << p$, we have achieved considerable dimension reduction. \subsection{Applying Matrix Partitioning} Using the approach of Section \ref{partmat}, partition $U^{(s)}$ into its columns, \begin{equation} U^{(s)} = (U_1,...,U_s) \end{equation} and thus write \begin{equation} \label{rqu1} R = Q U^{(s)} = Q (U_1,...,U_s) \end{equation} After that second equality, pretend that $Q$ and the $U_i$ are ``numbers.'' Then the last expression in (\ref{rqu1}), \begin{equation} Q (U_1,...,U_s) \end{equation} is a ``scalar'' times a ``vector,'' and is thus equal to \begin{equation} (QU_1,...,QU_s) \end{equation} So, the $i^{th}$ column of $R$ is $Q U_i$. The latter quantity is of the $Ax$, matrix-times-vector form of Section \ref{mfact}, so it is a linear combination of the columns of $Q$, with the coefficients in that linear combination being the elements in $U_i$. Recall that each column of $Q$ is one variable; e.g.\ for people, there may be an age column, a height column, a weight column and so on. Each column in $R$ is one of our new variables. Therefore: \begin{quote} The $i^{th}$ new variable is equal to a linear combination of the old variables. \end{quote} So, a new variable might be, say, 1.9 age + 0.3 height + 1.2 weight. % \subsection{Why It Works} % % Recall that the new variables have the $\lambda_i$ for variances and are % uncorrelated. Here's why: % % It is common (especially in neural networks, by the way) to % \textit{center} and \textbf{scale} one's data, meaning to subtract from % each variable its mean, and divide by its standard deviation. For % instance, if one variable is human height and its standard deviation is % 3.2 inches, we divide the height column in $Q$ by 3.2 (after subtracting % mean height). % % The new mean and standard deviation are 0 and 1. In that case, % (\ref{corrdef}) becomes % % % \begin{equation} % \frac{1}{n} \sum_{i=1}^n x_i y_i % \end{equation} % % Using this insight, the reader should verify that with the matrices $Q$ % and $R$ above (old and new data), their correlation matrices are $Q'Q$ % and $R'R$. Let's see how that works out: % % \begin{eqnarray} % \label{rrqq} % R'R &=& (Q U^{(s)})' (QU^{(s)}) \\ % &=& % U^{(s)} {}' Q' ~ Q U^{(s)} \\ % \end{eqnarray} % % using the fact that $(AB)' = B'A'$. % % But remember that the $U_i$ are eigenvectors of the correlation matrix, % in this case $Q'Q$! So % % \begin{equation} % \label{qqu} % (Q'Q) U^{(s)} = (Q'Q) (U_1,U_2,...,U_s) = % (\lambda_1 U_1,...,\lambda_s U_s) % \end{equation} % % Finally, recall that the $U_i$ are orthogonal with length 1. % Substituting (\ref{qqu}) in (\ref{rrqq})e then get % % \begin{equation} % R'R = U^{(s)}'(Q'Q) U^{(s)} = % U^{(s)}'(\lambda_1 U_1,...,\lambda_s U_s) % = \textrm{diag}(\lambda_1,...,\lambda_s) % \end{equation} % % the matrix having the $\lambda_i$ on the diagonal and 0s elsewhere. The % latter imply that the new variables are uncorrelated, and the former % fact says that the variances of the new variables are the $\lambda_i$, % as claimed. \subsection{Choosing the Number of Principal Components} The number of components we use, $s$, is called a \textit{tuning parameter} or \textit{hyperparameter}. So, how do we choose $s$? This is the hard part, and there is no universal good method. Typically $s$ is chosen so that \begin{equation} \sum_{j=1}^s \lambda_j \end{equation} is ``most'' of total variance (again, that total is the above expression with $p$ instead of $s$), but even this is usually done informally. In ML/RS settings, though, $s$ is typically chosen by a technique called \textit{cross validation}, to be discussed in Chapter \ref{chap:infra2}. \subsection{Software and Example} The most commonly used R function for PCA is \textbf{prcomp()}. As with many R functions, it has many optional arguments; we'll take the default values here. For our example, let's use the Turkish Teaching Evaluation data, available from the UC Irvine Machine Learning Data Repository. It consists of 5820 student evaluations of university instructors. Each student evaluation consists of answers to 28 questions, each calling for a rating of 1-5, plus some other variables we won't consider here. \begin{lstlisting} > turk <- read.csv('turkiye-student-evaluation.csv',header=T) > head(turk) instr class nb.repeat attendance difficulty Q1 Q2 Q3 Q4 1 1 2 1 0 4 3 3 3 3 2 1 2 1 1 3 3 3 3 3 3 1 2 1 2 4 5 5 5 5 4 1 2 1 1 3 3 3 3 3 5 1 2 1 0 1 1 1 1 1 6 1 2 1 3 3 4 4 4 4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q15 Q16 Q17 Q18 Q19 1 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 4 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 5 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 6 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 Q20 Q21 Q22 Q23 Q24 Q25 Q26 Q27 Q28 1 3 3 3 3 3 3 3 3 3 2 3 3 3 3 3 3 3 3 3 3 5 5 5 5 5 5 5 5 5 4 3 3 3 3 3 3 3 3 3 5 1 1 1 1 1 1 1 1 1 6 4 4 4 4 4 4 4 4 4 > tpca <- prcomp(turk[,-(1:5)] \end{lstlisting} Let's explore the output. First, the standard deviations of the new variables: \begin{lstlisting} > tpca$sdev [1] 6.1294752 1.4366581 0.8169210 0.7663429 0.6881709 [6] 0.6528149 0.5776757 0.5460676 0.5270327 0.4827412 [11] 0.4776421 0.4714887 0.4449105 0.4364215 0.4327540 [16] 0.4236855 0.4182859 0.4053242 0.3937768 0.3895587 [21] 0.3707312 0.3674430 0.3618074 0.3527829 0.3379096 [26] 0.3312691 0.2979928 0.2888057 > tmp <- cumsum(tpca$sdev^2) > tmp / tmp[28] [1] 0.8219815 0.8671382 0.8817389 0.8945877 0.9049489 [6] 0.9142727 0.9215737 0.9280977 0.9341747 0.9392732 [11] 0.9442646 0.9491282 0.9534589 0.9576259 0.9617232 [16] 0.9656506 0.9694785 0.9730729 0.9764653 0.9797855 [21] 0.9827925 0.9857464 0.9886104 0.9913333 0.9938314 [26] 0.9962324 0.9981752 1.0000000 \end{lstlisting} This is striking. The first principal component (PC) already accounts for 82\% of the total variance among all 28 questions. The first five PCs cover over 90\%. This suggests that the designer of the evaluation survey could have written a much more concise survey instrument with almost the same utility. Now keep in mind that each PC here is essentially a ``super-question'' capturing student opinion via a weighted sum of the original 28 questions. Let's look at the first two PCs' weights: \begin{lstlisting} > tpca$rotation[,1] Q1 Q2 Q3 Q4 Q5 -0.1787291 -0.1869604 -0.1821853 -0.1841701 -0.1902141 Q6 Q7 Q8 Q9 Q10 -0.1870812 -0.1878324 -0.1867865 -0.1823915 -0.1923626 Q11 Q12 Q13 Q14 Q15 -0.1866948 -0.1862382 -0.1922729 -0.1911814 -0.1902380 Q16 Q17 Q18 Q19 Q20 -0.1962885 -0.1808833 -0.1935788 -0.1927359 -0.1931985 Q21 Q22 Q23 Q24 Q25 -0.1911060 -0.1908591 -0.1948393 -0.1931334 -0.1888957 Q26 Q27 Q28 -0.1908694 -0.1897555 -0.1886699 \end{lstlisting} \begin{lstlisting} > tpca$rotation[,2] Q1 Q2 Q3 Q4 Q5 0.35645673 0.23223504 0.11551155 0.24533527 0.20717759 Q6 Q7 Q8 Q9 Q10 0.20075314 0.24290761 0.24901577 0.12919618 0.18911720 Q11 Q12 Q13 Q14 Q15 0.11051480 0.21203229 -0.10616030 -0.15629705 -0.15533847 Q16 Q17 Q18 Q19 Q20 -0.04865706 -0.26259518 -0.12905840 -0.15363392 -0.19670071 Q21 Q22 Q23 Q24 Q25 -0.22007368 -0.22347198 -0.10278122 -0.06210583 -0.20787213 Q26 Q27 Q28 -0.12045026 -0.07204024 -0.21401477 \end{lstlisting} The first PC turned out to place approximately equal weights on all 28 questions. The second PC, though, placed its heaviest weight on Q1, with substantially varying weights on the other questions. While we are here, let's check that the columns of $U$ are orthogonal. \begin{lstlisting} > t(tpca$rotation[,1]) %*% tpca$rotation[,2] [,1] [1,] -2.012279e-16 \end{lstlisting} Yes, 0 (with roundoff error). As an exercise in matrix partitioning, the reader should run \begin{lstlisting} t(tpca$rotation) %*% tpca$rotation \end{lstlisting} then check that it produces the identity matrix $I$, then ponder why this should be the case. \subsection{More on the PC Coefficients} \label{coors} There is more to consider. Do the PC coefficients have any interpretation? The answer is probably no for ordinary people, but for the \textit{domain experts}, very possibly yes. In the teaching evaluation example above, a specialist in survey design or teaching methods may well be able to interpret the dominance of Q1 in the second PC. A method called \textit{factor analysis}, an extension of PCA, is popular in social science research. For the rest of us, PCA is just a handy way to do dimension reduction. But there is geometric terminology that will be helpful, as follows. Let's look at the \textbf{mlb} dataset from the \textbf{regtools} package. This is data on Major League baseball players. \begin{lstlisting} Name Team Position Height Weight Age 1 Adam_Donachie BAL Catcher 74 180 22.99 2 Paul_Bako BAL Catcher 74 215 34.69 3 Ramon_Hernandez BAL Catcher 72 210 30.78 4 Kevin_Millar BAL First_Baseman 72 210 35.43 5 Chris_Gomez BAL First_Baseman 73 188 35.71 6 Brian_Roberts BAL Second_Baseman 69 176 29.39 PosCategory 1 Catcher 2 Catcher 3 Catcher 4 Infielder 5 Infielder 6 Infielder \end{lstlisting} Let's apply PCA: \begin{lstlisting} > hw <- as.matrix(mlb[,4:5]) > pcout <- prcomp(hw) > pcout$rotation PC1 PC2 Height -0.05948695 0.99822908 Weight -0.99822908 -0.05948695 \end{lstlisting} If we were to plot \textbf{hw}, we would put \textbf{hw[1,]} at the point (74,1.0) on our graph. Recall from high school math that 74 and 180 are called the \textit{coordinates} of \textbf{co2[1,]}, with respect to our ``H axis'' and ``W axis.'' But in doing PCA, we are creating new axes, PC1 and PC2, which are rotated versions of the H and W axes. (Hence the naming of the $U$ matrix as ``rotation'' in the \textbf{prcomp()} return value.) Let's find the coordinates of \textbf{hw[1,]} with respect to the new axes: \begin{lstlisting} > hw[1,] %*% pcout$rotation PC1 PC2 [1,] -184.0833 63.1613 \end{lstlisting} So (74,180) has become (-184.1,63.2) under the new coordinate system. Let's see what the angle of rotation is. We can do that by seeing where a point on the H axis rotates to. \begin{lstlisting} > pc10 <- c(1,0) %*% pcout$rotation > pc10 PC1 PC2 [1,] -0.05948695 0.9982291 > (atan(pc10[2] / pc10[1])) * 180/pi [1] -86.58964 \end{lstlisting} Almost 90 degrees clockwise. \subsection{Scaling} Some analysts prefer to \textit{scale} the data before applying PCA. For each column, we would subtract the column mean and divide by hte column standard deviation. The column would now have mean 0.0 and variance 1.0. The rationale for doing this is that if PCA is applied to the original data, variables with large variance will dominate. And then units would play a role; e.g.\ a distance variable would have more impact if it were measured in kilometers than miles. Scaling does solve this problem, but its propriety is questionable. Consider a setting with two features, $A$ and $B$, with variances 500 and 2, respectively. Let $A'$ and $B'$ denote these features after centering and scaling. As noted, PCA is all about removing features with small variance, as they are essentially constant. If we work with $A$ and $B$, we would of course use only $A$. But if we work with $A'$ and $B'$, we would use both of them, as they both have variance 1.0. So, dealing with the disparate-variance problem (e.g. miles vs.\ kilometers) shouldn't generally be solved by ordinary scaling, i.e.\ by dividing by the standard deviation. An alternative is to divide each column by its mean. This addresses the miles-vs.-kilometers problem, and makes sense in that a variance is large or small in relation to its mean. \section{Vector Norms} \label{vecnorms} In math, the $l_p$ \textit{norm} of a vector in $n$-dimensional space is defined by \begin{equation} ||x||_p = \left ( \sum_{i=1}^n [x_i|^p \right )^{1/p} \end{equation} This is actually a family of norms, for $1 \leq p \leq \infty$.\footnote{$||x||_{\infty} = \max_i |x_i|$} You are familiar with the Euclidean norm, $p = 2$. In statistics and machine learning, we are often minimizing the norm of some vector, usually with either $p = 1$ or $p = 2$. You will often see the notation $L_p$ instead of $l_p$. However, in math the former is used for function spaces, while the latter designates $n$-dimensional vectors, and we use the latter here. You will also see references to the {\it Frobenius} norm of an $r \times s$ matrix. That is actually just the $l_2$ norm, treating the matrix as an $rs$-dimensional vector. \section{Handy Facts} \begin{itemize} \item For conformable matrices $A$ and $B$, \begin{equation} (AB)' = B'A' \end{equation} \item For invertible and conformable matrices $A$ and $B$, \begin{equation} (AB)^{-1} = B^{-1} A^{-1} \end{equation} \item The R functions \textbf{t()} and \textbf{solve()} find the transpose and inverse of their matrix arguments. A more numerically stable function for inversion is \textbf{qr()}. \end{itemize}