Rao–Blackwell theorem

Remarks

Sufficiency means that the conditional law of $X$ given $T$ is the same for all $\theta$, so $\delta_1$ is computed without knowing $\theta$. The tower property gives the same mean, and the law of total variance gives $\operatorname{Var}_\theta(\delta) = \operatorname{Var}_\theta(\delta_1) + E_\theta[\operatorname{Var}(\delta \mid T)]$. The last term is nonnegative, and it is zero exactly when $\delta$ is already a function of $T$ almost surely. The same argument with the conditional Jensen inequality shows that $\delta_1$ is at least as good as $\delta$ for every loss that is convex in the estimate.

Example: let $X_1, \dots, X_n$ be independent Bernoulli variables with parameter $p$. The estimator $\delta = X_1$ is unbiased for $p$, and $T = X_1 + \cdots + X_n$ is sufficient. By symmetry $E[X_1 \mid T] = T/n$, so the Rao–Blackwell step turns $X_1$, of variance $p(1-p)$, into the sample mean, of variance $p(1-p)/n$.

Sufficiency is needed: conditioning on a statistic that is not sufficient gives a quantity that depends on the unknown $\theta$ in general, and so it is not an estimator.

Going deeper

For a student

The theorem is the main tool for improving unbiased estimators: conditioning on a sufficient statistic keeps the bias and lowers the variance. Combined with completeness of the sufficient statistic, it gives the Lehmann–Scheffé theorem, which identifies the unique minimum-variance unbiased estimator. In practice one often guesses a crude unbiased estimator, such as a single observation or an indicator, and computes its conditional expectation given the sufficient statistic.