Cramér–Rao bound

Remarks

Let $s$ be the score. Differentiating the two integrals gives $E_\theta[s] = 0$ and $E_\theta[T s] = \psi'(\theta)$, so $\operatorname{Cov}_\theta(T, s) = \psi'(\theta)$. The Cauchy–Schwarz inequality then gives $\psi'(\theta)^2 \le \operatorname{Var}_\theta(T) \operatorname{Var}_\theta(s) = \operatorname{Var}_\theta(T) \, I(\theta)$. For $n$ independent observations the information is $n I(\theta)$, so the bound becomes $\psi'(\theta)^2 / (n I(\theta))$. In dimension $d$, the bound reads $\operatorname{Cov}_\theta(T) \succeq D \, I(\theta)^{-1} D^{\mathsf T}$, with $D$ the Jacobian matrix of $\psi$.

Example: for $n$ independent normal observations with mean $\theta$ and known variance $\sigma^2$, one has $I(\theta) = 1/\sigma^2$. The sample mean is unbiased with variance $\sigma^2/n = 1/(n I(\theta))$, so it attains the bound.

The fixed support is necessary. For $n$ independent uniform observations on $(0, \theta)$, the unbiased estimator $\frac{n+1}{n} \max_i X_i$ has variance $\theta^2 / (n(n+2))$. This is smaller than $\theta^2/n$, the value of the formal bound with $E_\theta[s^2] = 1/\theta^2$.

Going deeper

For a student

The bound is the benchmark for the efficiency of estimators: the ratio of the bound to the actual variance measures how much information an estimator wastes. It is attained exactly only in special cases, essentially for exponential families and the natural statistic. Asymptotically, under regularity conditions, the maximum likelihood estimator has variance close to $1/(n I(\theta))$ and so reaches the bound in the limit. The regularity conditions matter: models with a moving support, such as the uniform family, can be estimated faster than the bound allows.