Weighted least squares

Builds on Generalized least squares Ordinary least squares Heteroscedasticity

Definition

Let $X$ be a real $n \times p$ matrix with rows $x_1^T, \dots, x_n^T$ and rank $p$ , let $y \in \R^n$ , and let $w_1, \dots, w_n > 0$ be weights. Let $W$ be the diagonal matrix with diagonal entries $w_1, \dots, w_n$ . The weighted least squares estimator is the minimiser of $\sum_{i=1}^n w_i (y_i - x_i^T b)^2$ over $b \in \R^p$ , which is $\hat\beta_{\mathrm{WLS}} = (X^T W X)^{-1} X^T W y.$

Remarks

Example: take $p = 1$ and $X$ the column of ones, so that $x_i^T b = b$ . The function $\sum_i w_i (y_i - b)^2$ has derivative $-2 \sum_i w_i (y_i - b)$ , which vanishes at $b = \sum_i w_i y_i / \sum_i w_i$ , the weighted mean. The formula agrees: $X^T W X = \sum_i w_i$ and $X^T W y = \sum_i w_i y_i$ .

Non-example: if some weight is negative, the objective need not be bounded below. With $n = 1$ , $p = 1$ , $x_1 = 1$ and $w_1 = -1$ , the function $-(y_1 - b)^2$ has no minimum.

In a model $y = X\beta + \varepsilon$ with uncorrelated errors and $\operatorname{Var}(\varepsilon_i) = \sigma_i^2$ , the choice $w_i = 1/\sigma_i^2$ gives the generalized least squares estimator with $\Omega = \operatorname{diag}(\sigma_1^2, \dots, \sigma_n^2)$ . This is the usual remedy for known heteroscedasticity. With all weights equal it is ordinary least squares.

Reading path

To understand this page from the ground up, read these 4 pages in order. The foundations come first.

Show the path as a graph

Each page uses the pages above it. The colour of the bar on a box shows how strong the evidence is, and the box also gives it as text. The list has the same pages.

no evidence level

Tag	`NA5X6JVK5J`
URI	`m2://NA5X6JVK5J`
Kind	definition
Writing	W0 stub
MSC	62J05
Defines	Weighted least squares
Uses	Generalized least squares, Ordinary least squares, Heteroscedasticity

Weighted least squares

Definition

Remarks

Reading path

Further reading

What this page needs for W1

What links here