Ordinary least squares

Remarks

The minimisers are exactly the solutions of the normal equations $X^T X b = X^T y$. The fitted vector $\hat y = X \hat\beta = H y$, with $H = X (X^T X)^{-1} X^T$, is the orthogonal projection of $y$ onto the column space of $X$. So the residual vector $y - \hat y$ is orthogonal to every column of $X$.

Example: fit $y = b_0 + b_1 t$ to the points $(0, 0)$, $(1, 1)$, $(2, 3)$. Here $\bar t = 1$ and $\bar y = 4/3$, so $b_1 = \sum (t_i - \bar t)(y_i - \bar y) / \sum (t_i - \bar t)^2 = 3/2$ and $b_0 = \bar y - b_1 \bar t = -1/6$. Check: the residuals are $1/6$, $-1/3$, $1/6$, which sum to $0$ and are orthogonal to $(0, 1, 2)$. Non-example: if two columns of $X$ are equal, $X$ has rank less than $p$ and $X^T X$ is singular. Then $S$ has infinitely many minimisers, so the formula above does not apply and $\beta$ is not identified.