Statistics Sufficiency and information Theorem

Pitman–Koopman–Darmois theorem

Builds on Exponential family Sufficient statistic Independent and identically distributed random variables Probability density function

Statement

Let $\set{f_\theta : \theta \in \Theta}$ be a family of probability densities on $\R$ with the following properties.

There is an open interval $I$ , the same for all $\theta$ , such that $f_\theta(x) > 0$ for $x \in I$ and $f_\theta(x) = 0$ for $x \notin I$ .
Each $f_\theta$ is continuously differentiable on $I$ .

Let $n \ge 2$ and let $X_1, \dots, X_n$ be independent and identically distributed with density $f_\theta$ . Assume that there is a sufficient statistic $T = (T_1, \dots, T_k) \colon I^n \to \R^k$ for $\theta$ that is continuously differentiable, with $k < n$ . Then the family is an exponential family with at most $k$ terms: there are functions $h$ , $u_1, \dots, u_k$ , $\eta_1, \dots, \eta_k$ and $A$ such that $f_\theta(x) = h(x) \exp\Big( \sum_{i=1}^{k} \eta_i(\theta) u_i(x) - A(\theta) \Big), \qquad x \in I.$

Remarks

The converse is easy. For an exponential family, the sums $\sum_{j=1}^{n} u_i(X_j)$ , $i = 1, \dots, k$ , form a sufficient statistic of dimension $k$ for every sample size $n$ , by the factorization criterion. So, among families with a fixed support, exponential families are exactly those with a sufficient statistic whose dimension does not grow with $n$ .

The idea of the proof is to write the factorization of the joint density and take logarithms. Differentiating in the $x_j$ shows that $\partial_x \log f_\theta(x)$ lies, as a function of $\theta$ , in a space of dimension at most $k$ . Integrating in $x$ gives the exponential form.

The fixed support is necessary. For the uniform densities on $(0, \theta)$ , the maximum $\max_j X_j$ is sufficient and one-dimensional for every $n$ . Yet the support moves with $\theta$ , and this family is not an exponential family.

Going deeper

For a student

The theorem explains why exponential families appear everywhere in classical parametric statistics. Sufficient statistics of fixed dimension give conjugate priors, complete statistics and explicit minimum-variance unbiased estimators. The result is named after Pitman, Koopman and Darmois. The precise regularity conditions differ from one textbook to another, but some smoothness of the sufficient statistic is always needed to exclude pathological maps from $\R^n$ to $\R^k$ .

Reading path

To understand this page from the ground up, read these 5 pages in order. The foundations come first.

Show the path as a graph

Each page uses the pages above it. The colour of the bar on a box shows how strong the evidence is, and the box also gives it as text. The list has the same pages.

no evidence level

Exponential family concept
Independent and identically distributed random variables concept
Probability density function concept
Sufficient statistic concept
Pitman–Koopman–Darmois theorem statement

Tag	`SSFSDQX8RR`
URI	`m2://SSFSDQX8RR`
Kind	statement, theorem
Writing	W0 stub
MSC	62B05, 62E10
Status	This copy of the site has no build data. Run `m2 status SSFSDQX8RR`.
About	Exponential family, Sufficient statistic
Uses	Exponential family, Sufficient statistic, Independent and identically distributed random variables, Probability density function