Asymptotic normality of the maximum likelihood estimator

Remarks

Idea of proof: with probability tending to one, $\hat{\theta}_n$ is an interior maximizer, so the score equation $\sum_i s(\hat{\theta}_n; X_i) = 0$ holds. A Taylor expansion around $\theta_0$ gives $0 = \sum_i s(\theta_0; X_i) + H_n (\hat{\theta}_n - \theta_0)$, where $H_n$ is the Hessian of the log-likelihood at intermediate points. Conditions 2 and 3 allow differentiation under the integral, so the score has mean $0$ and covariance $I(\theta_0)$, and the central limit theorem gives $n^{-1/2}\sum_i s(\theta_0; X_i) \xrightarrow{d} \mathcal{N}_k(0, I(\theta_0))$. Condition 5 and consistency give $H_n/n \to -I(\theta_0)$ in probability, and solving for $\sqrt{n}(\hat{\theta}_n - \theta_0)$ gives the result. Consistency itself is provided, for example, by the consistency theorem.

Example: for i.i.d. Exponential($\lambda$) data, $\hat{\lambda}_n = 1/\bar{X}_n$ and $I(\lambda) = 1/\lambda^2$, so $\sqrt{n}(\hat{\lambda}_n - \lambda) \xrightarrow{d} \mathcal{N}(0, \lambda^2)$.

The interior condition is needed. For $\mathcal{N}(\mu, 1)$ data with $\Theta = [0, \infty)$ and $\mu_0 = 0$, the MLE is $\max(\bar{X}_n, 0)$. Then $\sqrt{n}\,\hat{\mu}_n$ converges to the law of $\max(Z, 0)$ with $Z$ standard normal, which is not normal. Smoothness in $\theta$ with a fixed support is also needed: for Uniform$(0, \theta)$ data the MLE is $\max_i X_i$, and $n(\theta_0 - \hat{\theta}_n)$ has an exponential limit.

Going deeper

For a student

The theorem justifies the usual Wald confidence intervals $\hat{\theta}_n \pm 1.96\, (n\, I(\hat{\theta}_n))^{-1/2}$ and the Wald test. Together with the delta method it gives limit laws for smooth functions of the MLE. The same expansion of the log-likelihood leads to Wilks' theorem for the likelihood-ratio statistic and to the score test. The modern view puts the result in the frame of local asymptotic normality, where the bound $I(\theta_0)^{-1}$ appears in the Hájek–Le Cam convolution theorem.