Bias–variance decomposition

Remarks

Proof: write $Y - \hat{f} = \varepsilon + (f - E\hat{f}) + (E\hat{f} - \hat{f})$. Expand the square and take expectations: the three cross terms vanish, because $E\varepsilon = 0$, $E(E\hat{f} - \hat{f}) = 0$, and $\varepsilon$ is independent of $\hat{f}$. The remaining terms are $\sigma^2$, the squared bias and the variance of $\hat{f}$.

The case without noise is the decomposition of the mean squared error of an estimator $\hat{\theta}$ of a parameter $\theta$: $$E\big[(\hat{\theta} - \theta)^2\big] = \operatorname{Bias}(\hat{\theta})^2 + \operatorname{Var}(\hat{\theta}),$$ with the bias $\operatorname{Bias}(\hat{\theta}) = E(\hat{\theta}) - \theta$.

Example: for i.i.d. $X_1, \dots, X_n$ with variance $\sigma^2$, the estimator $\hat{\sigma}^2 = \frac{1}{n}\sum_i (X_i - \bar{X})^2$ has bias $-\sigma^2/n$. For normal data its variance is $2(n-1)\sigma^4/n^2$, so its mean squared error is $(2n - 1)\sigma^4/n^2$, which is smaller than the mean squared error $2\sigma^4/(n-1)$ of the unbiased sample variance. A small bias can thus buy a larger reduction of variance.

The irreducible term $\sigma^2$ does not depend on $\hat{f}$; no predictor can do better than it on average.

Going deeper

For a student

The decomposition is the basis of the bias–variance tradeoff in model selection: adding complexity lowers the bias but raises the variance, and the best expected error lies in between. Shrinkage methods such as ridge regression accept bias to reduce variance. In point estimation it explains why an unbiased estimator need not have the smallest mean squared error. Note that the decomposition holds at each fixed point $x$; averaging over $x$ gives the decomposition of the expected prediction error.