Shannon–McMillan–Breiman theorem

Remarks

The set $\mathcal{P}_n(x)$ consists of the points whose first $n$ positions $x, Tx, \dots, T^{n-1}x$ lie in the same sets of $\mathcal{P}$ as those of $x$. The function $-\log \mu(\mathcal{P}_n(x))$ is the information carried by these $n$ symbols, and its integral is $H(\bigvee_{i=0}^{n-1} T^{-i}\mathcal{P})$. So the convergence in $L^1$ refines the definition of $h(T, \mathcal{P})$, and the theorem says that the information per symbol is almost surely constant. The proof writes the information as a Birkhoff sum of conditional information functions, plus an error, and applies Birkhoff's ergodic theorem together with the martingale convergence theorem. Shannon proved convergence in probability for ergodic Markov sources, McMillan proved $L^1$ convergence for stationary ergodic sources, and Breiman proved almost sure convergence. Example: for the Bernoulli scheme $B(p_1, \dots, p_k)$ with the partition by the zeroth coordinate, $\mu(\mathcal{P}_n(x)) = p_{x_0} \cdots p_{x_{n-1}}$, and the theorem is the strong law of large numbers for the i.i.d. variables $-\log p_{x_i}$, whose mean is $-\sum p_i \log p_i$. Without ergodicity the limit still exists almost everywhere, but it is an invariant function whose integral is $h(T, \mathcal{P})$.

Going deeper

For a student

This is the asymptotic equipartition property, the basis of data compression: about $nh$ nats, or $nh / \log 2$ bits, suffice to encode a typical message of length $n$. In ergodic theory it gives a local, pointwise meaning to the Kolmogorov–Sinai entropy. It is a key tool in the proofs of the Ornstein isomorphism theorem and of Sinai's factor theorem, and it has analogues for actions of amenable groups.