Stein's unbiased risk estimate

Remarks

Expand $\norm{X - \theta + g(X)}^2 = \norm{X - \theta}^2 + \norm{g(X)}^2 + 2 \sum_i (X_i - \theta_i) g_i(X)$. The first term has expectation $p\sigma^2$. By the multivariate Stein's lemma, $E_\theta[(X_i - \theta_i) g_i(X)] = \sigma^2 E_\theta[\partial_i g_i(X)]$, which gives the formula.

The point is that $\mathrm{SURE}(X)$ does not involve the unknown $\theta$. So it can be computed from the data and minimized over a family of estimators, for example to choose a threshold or a smoothing parameter.

Example: the James–Stein estimator with $\sigma = 1$ corresponds to $g(x) = -(p - 2) x / \norm{x}^2$. Then $\mathrm{SURE}(X) = p - (p - 2)^2 / \norm{X}^2$, whose expectation is the risk of $\delta_{JS}$. It is below $p$ for every $X$, which shows that $\delta_{JS}$ has risk below $p$ when $p \ge 3$.

Going deeper

For a student

The estimate is the standard tool to prove that shrinkage estimators beat the raw observation, as in Stein's example. It is also used in practice to select tuning parameters, for instance the threshold in wavelet shrinkage, as an alternative to cross-validation. The derivative term $\sum_i \partial_i g_i$ is the divergence of $g$. The expected divergence of $\delta$, which is $p$ plus the expected divergence of $g$, is the effective number of degrees of freedom used by the estimator.