---
tag: 54387XJ9QW
kind: statement
role: theorem
name: Consistency of the maximum likelihood estimator
msc: [62F12, 62F10]
uses: [0FREPFKNS7, 05FSTXM92W, 0XJVF81QF6, 0AJN7WHBJZ, 00R0X1WGPX, 0EFDDXTVD6, 605M5NMFD5, 0RWMRTPC87, 0BFDHF2H7N]
about: [0FREPFKNS7, 0XJVF81QF6]
sources:
  - url: https://en.wikipedia.org/wiki/Maximum_likelihood_estimation
    locator: Consistency
---

## Statement

Let $\set{f(\cdot; \theta) : \theta \in \Theta}$ be a family of probability densities with respect to a measure $\nu$, and let $X_1, X_2, \dots$ be [[i.i.d.|0AJN7WHBJZ]] with density $f(\cdot; \theta_0)$ for some $\theta_0 \in \Theta$.
Assume:

1. ([[identifiability|605M5NMFD5]]) if $\theta \ne \theta_0$, then $f(\cdot; \theta) \ne f(\cdot; \theta_0)$ on a set of positive $\nu$-measure;
2. $\Theta$ is a [[compact|00R0X1WGPX]] subset of $\R^k$;
3. for $\nu$-almost every $x$, the map $\theta \mapsto \log f(x; \theta)$ is continuous on $\Theta$;
4. $E_{\theta_0}\big[\sup_{\theta \in \Theta} \abs{\log f(X_1; \theta)}\big] < \infty$.

Let $\hat{\theta}_n$ be a measurable maximizer over $\Theta$ of the log-[[likelihood|05FSTXM92W]] $\ell_n(\theta) = \sum_{i=1}^n \log f(X_i; \theta)$.
Then $\hat{\theta}_n \to \theta_0$ in [[probability|0EFDDXTVD6]]: the [[maximum likelihood estimator|0FREPFKNS7]] is [[consistent|0XJVF81QF6]].

## Remarks

Idea of proof: let $L(\theta) = E_{\theta_0}[\log f(X_1; \theta)]$.
By [[Jensen's inequality|0RWMRTPC87]] for the strictly concave logarithm, and by identifiability, $L(\theta) < L(\theta_0)$ for every $\theta \ne \theta_0$.
Conditions 2 to 4 give a uniform [[law of large numbers|0BFDHF2H7N]]: $\sup_{\theta} \abs{\ell_n(\theta)/n - L(\theta)} \to 0$ in probability.
Since $L$ is continuous on the compact set $\Theta$, it stays below $L(\theta_0)$ by a fixed margin outside each neighbourhood of $\theta_0$.
So the maximizer of $\ell_n/n$ must eventually lie inside that neighbourhood.

Example: for i.i.d. $\mathcal{N}(\mu, 1)$ data with $\mu \in [-K, K]$, the conditions hold and $\hat{\mu}_n$ is the sample mean clipped to $[-K, K]$, which tends to $\mu_0$.

Identifiability is necessary: if $f(\cdot; \theta_1) = f(\cdot; \theta_0)$ for some $\theta_1 \ne \theta_0$, the data cannot distinguish $\theta_0$ from $\theta_1$, and no estimator converges to $\theta_0$ under both.
Compactness can be replaced by other conditions, such as concavity of $\theta \mapsto \log f(x; \theta)$, but some condition of this kind is needed.

## Explanations

### L1 human

With more and more data, the parameter value that makes the data most likely gets closer and closer to the true value.
The reason is that, on average, the true parameter gives the data a higher log-likelihood than any other parameter.
The conditions make sure that this average behaviour is reached uniformly over all parameter values.

### L2 human

This is Wald's approach to consistency, and the same argument proves consistency of general M-estimators and extremum estimators.
Consistency is the first step in proving the asymptotic normality of the maximum likelihood estimator, which needs a consistent estimator to linearize the likelihood equations around $\theta_0$.
The limit $L(\theta_0) - L(\theta)$ is the Kullback–Leibler divergence of $f(\cdot; \theta)$ from $f(\cdot; \theta_0)$, so maximum likelihood can be seen as minimizing an empirical version of this divergence.