---
tag: 3KZW2930V2
kind: concept
name: Ridge regression
msc: [62J07, 62J05]
sources:
  - url: https://en.wikipedia.org/wiki/Ridge_regression
same_as:
  wikidata: Q106091291
  wikipedia: Ridge regression
---

## Summary

Ridge regression estimates the coefficients of a linear model by least squares with a penalty on the squared Euclidean norm of the coefficients.
The penalty shrinks the coefficients towards zero.
It makes the estimate unique and stable even when the regressors are nearly collinear or more numerous than the observations.
It is a [[shrinkage estimator|GPBSS1V3SS]]: it accepts some bias in exchange for a smaller variance.
In numerical analysis the same idea is called Tikhonov regularization.

## Explanations

### L1 human

When two regressors are almost the same, [[ordinary least squares|YM708KHV2W]] can give huge coefficients of opposite signs that nearly cancel.
Ridge regression adds a cost for large coefficients, so such wild solutions are no longer worth it.
A tuning parameter $\lambda$ sets the strength of the cost: $\lambda = 0$ is ordinary least squares and a very large $\lambda$ pushes all coefficients to zero.
With a single regressor and no intercept, the ridge slope is $\sum_i x_i y_i / (\sum_i x_i^2 + \lambda)$, a shrunk version of the least squares slope.
Unlike the [[lasso|KZ226116HW]], ridge regression does not set coefficients exactly to zero.