Statistics Linear inference and regression Theorem

Gauss–Markov theorem

Builds on General linear model Ordinary least squares Covariance matrix Bias of an estimator Variance Positive semidefinite matrix Heteroscedasticity

Statement

Let $X$ be a fixed real $n \times p$ matrix of rank $p$ , and let $\beta \in \R^p$ be an unknown parameter. Let $\varepsilon$ be a random vector in $\R^n$ and $y = X\beta + \varepsilon.$ Assume the following.

$E[\varepsilon] = 0$ .
$\operatorname{Cov}(\varepsilon) = \sigma^2 I_n$ for some unknown $\sigma^2 > 0$ , so the errors are uncorrelated and homoscedastic.

Let $\hat\beta = (X^T X)^{-1} X^T y$ be the ordinary least squares estimator. Let $\tilde\beta = C y$ be any estimator with $C$ a fixed $p \times n$ matrix such that $E[\tilde\beta] = \beta$ for every $\beta \in \R^p$ . Then $\hat\beta$ is unbiased, and the matrix $\operatorname{Cov}(\tilde\beta) - \operatorname{Cov}(\hat\beta)$ is positive semidefinite. In particular, $\operatorname{Var}(c^T \tilde\beta) \ge \operatorname{Var}(c^T \hat\beta)$ for every $c \in \R^p$ .

Remarks

Write $C = (X^T X)^{-1} X^T + D$ . Unbiasedness for every $\beta$ forces $D X = 0$ . Then $\operatorname{Cov}(\tilde\beta) = \sigma^2 C C^T = \sigma^2 (X^T X)^{-1} + \sigma^2 D D^T$ , because the cross terms contain $D X$ . The matrix $D D^T$ is positive semidefinite, which gives the claim.

For the model $y_i = \mu + \varepsilon_i$ , the estimator $\hat\beta$ is the sample mean. Here the theorem says that the sample mean has the smallest variance among all weighted averages $\sum_i c_i y_i$ with $\sum_i c_i = 1$ .

The errors need not be normal or independent. If the covariance matrix of $\varepsilon$ is not a multiple of $I_n$ , then ordinary least squares is still unbiased but in general no longer best. The theorem compares only linear unbiased estimators: a biased estimator, such as a ridge estimator, can have a smaller mean squared error.

Going deeper

For a student

The hypotheses are only about the first two moments of the errors: mean zero, equal variances and no correlation. When the errors are also normal, a stronger result holds: least squares is then the minimum-variance unbiased estimator among all unbiased estimators, linear or not. When the variances differ or the errors are correlated with a known covariance structure, one transforms the model to restore the hypotheses. That leads to generalized and weighted least squares. The theorem says nothing about biased estimators, and shrinkage methods trade a small bias for a large drop in variance.

Reading path

To understand this page from the ground up, read these 8 pages in order. The foundations come first.

Show the path as a graph

Each page uses the pages above it. The colour of the bar on a box shows how strong the evidence is, and the box also gives it as text. The list has the same pages.

no evidence level

Where this leads

1 page builds on this one.

Generalized least squares

Tag	`N3YMNMWANM`
URI	`m2://N3YMNMWANM`
Kind	statement, theorem
Writing	W0 stub
MSC	62J05
Status	This copy of the site has no build data. Run `m2 status N3YMNMWANM`.
About	Ordinary least squares, Linear regression
Uses	General linear model, Ordinary least squares, Covariance matrix, Bias of an estimator, Variance, Positive semidefinite matrix, Heteroscedasticity
Same as	Wikidata Q428134 Wikipedia Gauss–Markov theorem

Markdown · JSON · Edit on GitHub

What this page needs for W1

an inventory row: the row of the inventory that the node accounts for (0 of 1)
a source with a locator (0 of 1)

The writing levels say how complete the text of a page is. They do not measure its truth.

What links here

Used by

Generalized least squares