Bernstein–von Mises theorem

Builds on Parametric model Posterior probability Prior probability Fisher information Multivariate normal distribution Total variation distance between probability measures Convergence in probability Independent and identically distributed random variables

Statement

Let $\set{P_\theta : \theta \in \Theta}$ be a parametric model with $\Theta \subseteq \R^k$ open, and let $X_1, X_2, \dots$ be i.i.d. with law $P_{\theta_0}$ for some $\theta_0 \in \Theta$ . Assume the following.

The model is differentiable in quadratic mean at $\theta_0$ , with nonsingular Fisher information matrix $I(\theta_0)$ .
For every $\varepsilon > 0$ there are tests $\varphi_n = \varphi_n(X_1, \dots, X_n)$ with $E_{\theta_0}[\varphi_n] \to 0$ and $\sup_{\norm{\theta - \theta_0} \ge \varepsilon} E_\theta[1 - \varphi_n] \to 0$ .
The prior has a density with respect to Lebesgue measure that is continuous and positive in a neighbourhood of $\theta_0$ .

Let $\hat{\theta}_n$ be an efficient estimator, that is, $\sqrt{n}(\hat{\theta}_n - \theta_0) = \frac{1}{\sqrt{n}} \sum_{i=1}^{n} I(\theta_0)^{-1} \dot{\ell}_{\theta_0}(X_i) + o_P(1)$ , where $\dot{\ell}_{\theta_0}$ is the score function. Then the posterior distribution $\Pi_n$ of $\theta$ given $X_1, \dots, X_n$ satisfies $\norm{\Pi_n - \mathcal{N}\left(\hat{\theta}_n, \tfrac{1}{n} I(\theta_0)^{-1}\right)}_{TV} \to 0$ in $P_{\theta_0}$ -probability, where $\norm{\cdot}_{TV}$ is the total variation distance.

Remarks

The proof expands the log-likelihood near $\theta_0$ : by local asymptotic normality it is, up to $o_P(1)$ , a quadratic function of $\sqrt{n}(\theta - \theta_0)$ . The tests of hypothesis 2 show that the posterior mass outside shrinking neighbourhoods of $\theta_0$ vanishes, and hypothesis 3 makes the prior nearly constant on these neighbourhoods. So the posterior is close to the normalised exponential of a quadratic form, that is, to a multivariate normal law. Under the usual regularity conditions the maximum likelihood estimator is efficient and can serve as $\hat{\theta}_n$ . Example: for i.i.d. Bernoulli( $p_0$ ) data with $0 < p_0 < 1$ and any beta prior, the posterior of $p$ is approximately $\mathcal{N}(\hat{p}_n, \hat{p}_n(1 - \hat{p}_n)/n)$ for large $n$ , with $\hat{p}_n$ the proportion of successes. The prior must charge every neighbourhood of $\theta_0$ : a prior that vanishes near $\theta_0$ gives a posterior that stays away from $\theta_0$ , and the conclusion fails. The theorem can fail in nonparametric models.

Going deeper

For a student

The theorem is the bridge between Bayesian inference and frequentist asymptotics in regular parametric models. It implies that an equal-tailed $95\%$ credible interval for a coordinate of $\theta$ has frequentist coverage tending to $95\%$ . Under additional integrability conditions, the posterior mean is also asymptotically efficient, like the maximum likelihood estimator. Its hypotheses are the same local asymptotic normality conditions that drive the asymptotic theory of the maximum likelihood estimator.

Reading path

To understand this page from the ground up, read these 72 pages in order. The foundations come first.

Construction of the reals from Cauchy sequences concept
Element definition
Fisher information concept
Integers object
Ordered field definition
Proposition concept
Random experiment concept
Rational numbers object
Real numbers object
Total variation distance between probability measures concept
Set definition
Ring of integers statement
Conjunction definition
Disjunction definition
Implication definition
Predicate definition
Empty set object
Family of sets definition
Ordered pair definition
Intersection definition
Union definition
Sample space definition
The rationals form a field statement
Existential quantifier definition
Universal quantifier definition
Disjoint sets definition
Cartesian product definition
Subset definition
Union of a family of sets definition
The rationals form an ordered field statement
Uniqueness quantifier definition
Universal set definition
Binary relation definition
Partition of a set definition
Order on the reals definition
Power set definition
Complement definition
Function definition
Interval definition
Indicator function definition
Preimage of a set definition
Sigma-algebra definition
Event definition
Borel sigma-algebra definition
Probability axioms concept
Probability measure definition
Probability distribution definition
Dirac measure definition
Statistical model definition
Probability space definition
Conditional probability definition
Random variable definition
Parametric model definition
Cumulative distribution function definition
Law of a random variable definition
Convergence in probability definition
Independence of two events definition
Law of total probability statement
Random variable with a density definition
Identically distributed random variables definition
Bayes' theorem statement
Mutual independence of events definition
Probability density function definition
Prior probability concept
Independent sigma-algebras definition
Density of the normal distribution definition
Posterior probability concept
Independence of random variables definition
Normal distribution definition
Independent and identically distributed random variables definition
Multivariate normal distribution definition
Bernstein–von Mises theorem statement

Tag	`1E8QNK0Q92`
URI	`m2://1E8QNK0Q92`
Kind	statement, theorem
Writing	W0 stub
MSC	62F15, 62F12
Status	This copy of the site has no build data. Run `m2 status 1E8QNK0Q92`.
About	Posterior probability
Uses	Parametric model, Posterior probability, Prior probability, Fisher information, Multivariate normal distribution, Total variation distance between probability measures, Convergence in probability, Independent and identically distributed random variables
Same as	Wikidata Q4894580 Wikipedia Bernstein–von Mises theorem