---
tag: R2EHQ06B98
kind: definition
name: Principal component analysis
msc: [62H25, 15A18]
defines: ABXD13M340
uses: [0V7G6WMFK0, 0N6960W0BV, 0BQAA2BK3Q, 0S9GGTR0EX, 06J4PXRG22, 0477R9PYAE]
sources:
  - url: https://en.wikipedia.org/wiki/Principal_component_analysis
---

## Definition

Let $X$ be a random vector in $\R^p$ with finite second moments, mean $\mu$ and [[covariance matrix|0V7G6WMFK0]] $\Sigma$.
By the [[spectral theorem|06J4PXRG22]], $\Sigma$ has an [[orthonormal basis|0S9GGTR0EX]] of [[eigenvectors|0N6960W0BV]] $v_1, \ldots, v_p$ with [[eigenvalues|0BQAA2BK3Q]] $\lambda_1 \ge \lambda_2 \ge \cdots \ge \lambda_p \ge 0$.
The $k$-th *principal component* of $X$ is the random variable
$$Y_k = v_k^T (X - \mu), \qquad k = 1, \ldots, p.$$
The vector $v_k$ is the $k$-th principal direction, or loading vector.
*Principal component analysis* is the passage from $X$ to $(Y_1, \ldots, Y_p)$, usually followed by keeping only $Y_1, \ldots, Y_q$ for some $q < p$.
For a data matrix, the same construction is applied to the sample covariance matrix.

## Remarks

The components are uncorrelated and $\operatorname{Var}(Y_k) = \lambda_k$.
Equivalently, $v_1$ maximises the [[variance|0477R9PYAE]] $\operatorname{Var}(a^T X) = a^T \Sigma a$ over unit vectors $a$, and $v_k$ maximises it over unit vectors orthogonal to $v_1, \ldots, v_{k-1}$.
The principal directions are unique up to sign when the eigenvalues are distinct.

Example: let $\Sigma = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}$.
Its eigenvalues are $3$ and $1$, with unit eigenvectors $v_1 = (1, 1)/\sqrt{2}$ and $v_2 = (1, -1)/\sqrt{2}$.
Check: $v_1^T \Sigma v_1 = (2 + 1 + 1 + 2)/2 = 3$.
Non-example: the direction $a = (1, 0)$ gives $a^T \Sigma a = 2 < 3$, so $a^T (X - \mu)$ is not the first principal component.

The fraction $(\lambda_1 + \cdots + \lambda_q)/(\lambda_1 + \cdots + \lambda_p)$ is the proportion of the total variance kept by the first $q$ components.
For a centred data matrix, the principal directions are the right singular vectors of its singular value decomposition.