Delta method

Remarks

Idea of proof: by differentiability, $g(T_n) - g(\theta) = g'(\theta)(T_n - \theta) + r(T_n)(T_n - \theta)$ with $r(t) \to 0$ as $t \to \theta$. Since $T_n \to \theta$ in probability, $r(T_n) \to 0$ in probability, and $\sqrt{n}(T_n - \theta)$ is bounded in probability. So the remainder times $\sqrt{n}$ tends to $0$ in probability, and Slutsky's lemma gives the limit $g'(\theta)\,\mathcal{N}(0, \sigma^2)$.

Multivariate form: if $\sqrt{n}(T_n - \theta) \xrightarrow{d} \mathcal{N}_k(0, \Sigma)$ in $\R^k$, with a multivariate normal limit, and $g \colon \R^k \to \R^m$ is differentiable at $\theta$ with Jacobian matrix $J$, then $\sqrt{n}(g(T_n) - g(\theta)) \xrightarrow{d} \mathcal{N}_m(0, J \Sigma J^T)$.

Example: for i.i.d. Poisson variables with mean $\lambda > 0$, the central limit theorem gives $\sqrt{n}(\bar{X}_n - \lambda) \xrightarrow{d} \mathcal{N}(0, \lambda)$. With $g(t) = 2\sqrt{\abs{t}}$, $g'(\lambda) = 1/\sqrt{\lambda}$, so $\sqrt{n}\big(2\sqrt{\bar{X}_n} - 2\sqrt{\lambda}\big) \xrightarrow{d} \mathcal{N}(0, 1)$. The limit variance no longer depends on $\lambda$: the square root is a variance-stabilizing transformation for Poisson data.

If $g'(\theta) = 0$, the limit is the point mass at $0$, and one needs a second-order expansion: if $g''(\theta) \ne 0$ exists, then $n(g(T_n) - g(\theta)) \xrightarrow{d} \frac{1}{2} g''(\theta) \sigma^2 \chi^2_1$.

Going deeper

For a student

The delta method turns the central limit theorem for averages into limit laws for smooth functions of averages: ratios, logarithms, correlation coefficients, odds ratios. It gives approximate standard errors and confidence intervals for such quantities. It is also the step that passes from the asymptotic normality of an estimator $\hat{\theta}$ to that of $g(\hat{\theta})$, for example for maximum likelihood estimators. Variance-stabilizing transformations are chosen with it: one picks $g$ so that $g'(\theta)^2 \sigma^2(\theta)$ does not depend on $\theta$.