Math 2.0

Statistics Definition

Principal component analysis

Builds on Covariance matrixEigenvectorEigenvalueOrthonormal basisSpectral theoremVariance

Definition

Let XX be a random vector in ℝp\R^p with finite second moments, mean μ\mu and covariance matrix Σ\Sigma. By the spectral theorem, Σ\Sigma has an orthonormal basis of eigenvectors v1,…,vpv_1, \ldots, v_p with eigenvalues λ1≥λ2≥⋯≥λp≥0\lambda_1 \ge \lambda_2 \ge \cdots \ge \lambda_p \ge 0. The kk-th principal component of XX is the random variable Yk=vkT(X−μ),k=1,…,p.Y_k = v_k^T (X - \mu), \qquad k = 1, \ldots, p. The vector vkv_k is the kk-th principal direction, or loading vector. Principal component analysis is the passage from XX to (Y1,…,Yp)(Y_1, \ldots, Y_p), usually followed by keeping only Y1,…,YqY_1, \ldots, Y_q for some q<pq < p. For a data matrix, the same construction is applied to the sample covariance matrix.

Remarks

The components are uncorrelated and Var⁡(Yk)=λk\operatorname{Var}(Y_k) = \lambda_k. Equivalently, v1v_1 maximises the variance Var⁡(aTX)=aTΣa\operatorname{Var}(a^T X) = a^T \Sigma a over unit vectors aa, and vkv_k maximises it over unit vectors orthogonal to v1,…,vk−1v_1, \ldots, v_{k-1}. The principal directions are unique up to sign when the eigenvalues are distinct.

Example: let Σ=(2112)\Sigma = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}. Its eigenvalues are 33 and 11, with unit eigenvectors v1=(1,1)/2v_1 = (1, 1)/\sqrt{2} and v2=(1,−1)/2v_2 = (1, -1)/\sqrt{2}. Check: v1TΣv1=(2+1+1+2)/2=3v_1^T \Sigma v_1 = (2 + 1 + 1 + 2)/2 = 3. Non-example: the direction a=(1,0)a = (1, 0) gives aTΣa=2<3a^T \Sigma a = 2 < 3, so aT(X−μ)a^T (X - \mu) is not the first principal component.

The fraction (λ1+⋯+λq)/(λ1+⋯+λp)(\lambda_1 + \cdots + \lambda_q)/(\lambda_1 + \cdots + \lambda_p) is the proportion of the total variance kept by the first qq components. For a centred data matrix, the principal directions are the right singular vectors of its singular value decomposition.

Reading path

To understand this page from the ground up, read these 25 pages in order. The foundations come first.

Show the path as a graph
Covariance matrix, concept, no evidence levelCovariance matrixconceptEigenvalue, concept, no evidence levelEigenvalueconceptEigenvector, concept, no evidence levelEigenvectorconceptOrthonormal basis, concept, no evidence levelOrthonormal basisconceptVariance, concept, no evidence levelVarianceconceptVector space, definition, no evidence levelVector spacedefinitionInner product, definition, no evidence levelInner productdefinitionLinear combination, definition, no evidence levelLinear combinationdefinitionLinear map, definition, no evidence levelLinear mapdefinitionLinear subspace, definition, no evidence levelLinear subspacedefinitionInner product space, definition, no evidence levelInner product spacedefinitionOrthogonal vectors, definition, no evidence levelOrthogonal vectorsdefinitionLinear independence, definition, no evidence levelLinear independencedefinitionEndomorphism, definition, no evidence levelEndomorphismdefinitionLinear span, definition, no evidence levelLinear spandefinitionAdjoint of a linear map, definition, no evidence levelAdjoint of a linear m…definitionOrthonormal family, definition, no evidence levelOrthonormal familydefinitionEigenvalue, definition, no evidence levelEigenvaluedefinitionSpanning set, definition, no evidence levelSpanning setdefinitionSelf-adjoint operator, definition, no evidence levelSelf-adjoint operatordefinitionEigenvector, definition, no evidence levelEigenvectordefinitionBasis, definition, no evidence levelBasisdefinitionOrthonormal basis, definition, no evidence levelOrthonormal basisdefinitionSpectral theorem, statement, no evidence levelSpectral theoremstatementPrincipal component analysis, definition, no evidence levelPrincipal component a…definition
Each page uses the pages above it. The colour of the bar on a box shows how strong the evidence is, and the box also gives it as text. The list has the same pages.
  • no evidence level
  1. Covariance matrix concept
  2. Eigenvalue concept
  3. Eigenvector concept
  4. Orthonormal basis concept
  5. Variance concept
  6. Vector space definition
  7. Inner product definition
  8. Linear combination definition
  9. Linear map definition
  10. Linear subspace definition
  11. Inner product space definition
  12. Orthogonal vectors definition
  13. Linear independence definition
  14. Endomorphism definition
  15. Linear span definition
  16. Adjoint of a linear map definition
  17. Orthonormal family definition
  18. Eigenvalue definition
  19. Spanning set definition
  20. Self-adjoint operator definition
  21. Eigenvector definition
  22. Basis definition
  23. Orthonormal basis definition
  24. Spectral theorem statement
  25. Principal component analysis definition

Further reading

  1. Web https://en.wikipedia.org/wiki/Principal_component_analysis
About this page
TagR2EHQ06B98
URIm2://R2EHQ06B98
Kinddefinition
WritingW0 stub
MSC62H25, 15A18
DefinesPrincipal component analysis
UsesCovariance matrix, Eigenvector, Eigenvalue, Orthonormal basis, Spectral theorem, Variance

Markdown · JSON · Edit on GitHub

What this page needs for W1

  • an inventory row: the row of the inventory that the node accounts for (0 of 1)
  • a source with a locator (0 of 1)

The writing levels say how complete the text of a page is. They do not measure its truth.

What links here

No page links to this page yet.