Ridge regression

Remarks

The matrix $X^T X + \lambda I_p$ is positive definite for $\lambda > 0$, so the minimiser exists and is unique for every $X$, whatever its rank. The formula comes from setting the gradient $-2 X^T (y - X b) + 2 \lambda b$ to zero.

Example: take $p = 1$, $X = (1, 2)^T$, $y = (1, 2)^T$ and $\lambda = 1$. Then $X^T X = 5$ and $X^T y = 5$, so $\hat\beta_1 = 5/6$, while the ordinary least squares slope is $1$.

Non-example: with $\lambda = 0$ the problem is plain least squares, and if $X^T X$ is singular, for example when two columns of $X$ are equal, the minimiser is not unique.

In practice the regressors are usually standardised and the intercept is not penalised. For each $\lambda > 0$ there is $t > 0$ such that $\hat\beta_\lambda$ also minimises $\norm{y - Xb}^2$ under the constraint $\norm{b}^2 \le t$. In a normal linear model, $\hat\beta_\lambda$ is the posterior mode for a centred normal prior on $\beta$.