Consistency of the maximum likelihood estimator

Remarks

Idea of proof: let $L(\theta) = E_{\theta_0}[\log f(X_1; \theta)]$. By Jensen's inequality for the strictly concave logarithm, and by identifiability, $L(\theta) < L(\theta_0)$ for every $\theta \ne \theta_0$. Conditions 2 to 4 give a uniform law of large numbers: $\sup_{\theta} \abs{\ell_n(\theta)/n - L(\theta)} \to 0$ in probability. Since $L$ is continuous on the compact set $\Theta$, it stays below $L(\theta_0)$ by a fixed margin outside each neighbourhood of $\theta_0$. So the maximizer of $\ell_n/n$ must eventually lie inside that neighbourhood.

Example: for i.i.d. $\mathcal{N}(\mu, 1)$ data with $\mu \in [-K, K]$, the conditions hold and $\hat{\mu}_n$ is the sample mean clipped to $[-K, K]$, which tends to $\mu_0$.

Identifiability is necessary: if $f(\cdot; \theta_1) = f(\cdot; \theta_0)$ for some $\theta_1 \ne \theta_0$, the data cannot distinguish $\theta_0$ from $\theta_1$, and no estimator converges to $\theta_0$ under both. Compactness can be replaced by other conditions, such as concavity of $\theta \mapsto \log f(x; \theta)$, but some condition of this kind is needed.

Going deeper

For a student

This is Wald's approach to consistency, and the same argument proves consistency of general M-estimators and extremum estimators. Consistency is the first step in proving the asymptotic normality of the maximum likelihood estimator, which needs a consistent estimator to linearize the likelihood equations around $\theta_0$. The limit $L(\theta_0) - L(\theta)$ is the Kullback–Leibler divergence of $f(\cdot; \theta)$ from $f(\cdot; \theta_0)$, so maximum likelihood can be seen as minimizing an empirical version of this divergence.