{
  "aliases": null,
  "backlinks": null,
  "body": "\n## Statement\n\nLet $(X, \\mathcal{B}, \\mu)$ be a [[probability space|0NFPJVJJHX]] and let $T \\colon X \\to X$ be an [[ergodic|AZRPHS478B]] [[measure-preserving transformation|556WBDAHNP]].\nLet $\\mathcal{P}$ be a finite measurable [[partition|08QMKTWF9T]] of $X$.\nLet $h(T, \\mathcal{P})$ be the entropy of $T$ with respect to $\\mathcal{P}$, as in the definition of the [[Kolmogorov–Sinai entropy|X1D1C7VH6P]].\nFor $n \\ge 1$ and $x \\in X$, let $\\mathcal{P}_n(x)$ be the set of the partition $\\bigvee_{i=0}^{n-1} T^{-i}\\mathcal{P}$ that contains $x$.\nThen\n$$\\lim_{n \\to \\infty} -\\frac{1}{n} \\log \\mu(\\mathcal{P}_n(x)) = h(T, \\mathcal{P})$$\nfor [[almost every|RARTMA0ZEC]] $x \\in X$, and also in $L^1(\\mu)$.\n\n## Remarks\n\nThe set $\\mathcal{P}_n(x)$ consists of the points whose first $n$ positions $x, Tx, \\dots, T^{n-1}x$ lie in the same sets of $\\mathcal{P}$ as those of $x$.\nThe function $-\\log \\mu(\\mathcal{P}_n(x))$ is the information carried by these $n$ symbols, and its integral is $H(\\bigvee_{i=0}^{n-1} T^{-i}\\mathcal{P})$.\nSo the convergence in $L^1$ refines the definition of $h(T, \\mathcal{P})$, and the theorem says that the information per symbol is almost surely constant.\nThe proof writes the information as a Birkhoff sum of conditional information functions, plus an error, and applies Birkhoff's ergodic theorem together with the martingale convergence theorem.\nShannon proved convergence in probability for ergodic Markov sources, McMillan proved $L^1$ convergence for stationary ergodic sources, and Breiman proved almost sure convergence.\nExample: for the Bernoulli scheme $B(p_1, \\dots, p_k)$ with the partition by the zeroth coordinate, $\\mu(\\mathcal{P}_n(x)) = p_{x_0} \\cdots p_{x_{n-1}}$, and the theorem is the strong law of large numbers for the i.i.d. variables $-\\log p_{x_i}$, whose mean is $-\\sum p_i \\log p_i$.\nWithout ergodicity the limit still exists almost everywhere, but it is an invariant function whose integral is $h(T, \\mathcal{P})$.\n\n## Explanations\n\n### L1 human\n\nWatch a long stretch of output of a stationary random source, such as a long English text.\nThe theorem says that almost every long stretch of $n$ symbols has probability close to $e^{-nh}$, where $h$ is the entropy of the source.\nSo the likely messages all have about the same probability, and there are about $e^{nh}$ of them.\n\n### L2 human\n\nThis is the asymptotic equipartition property, the basis of data compression: about $nh$ nats, or $nh / \\log 2$ bits, suffice to encode a typical message of length $n$.\nIn ergodic theory it gives a local, pointwise meaning to the [[Kolmogorov–Sinai entropy|X1D1C7VH6P]].\nIt is a key tool in the proofs of the Ornstein isomorphism theorem and of Sinai's factor theorem, and it has analogues for actions of amenable groups.\n",
  "commit": "ff8a40749dc16545e3d2fc637f6b0196e81d0bfa",
  "front": {
    "about": [
      "X1D1C7VH6P"
    ],
    "kind": "statement",
    "msc": [
      "37A35",
      "28D20",
      "94A17"
    ],
    "name": "Shannon–McMillan–Breiman theorem",
    "role": "theorem",
    "sources": [
      {
        "url": "https://en.wikipedia.org/wiki/Asymptotic_equipartition_property"
      }
    ],
    "tag": "HK8339ECNN",
    "uses": [
      "0NFPJVJJHX",
      "556WBDAHNP",
      "AZRPHS478B",
      "X1D1C7VH6P",
      "RARTMA0ZEC",
      "08QMKTWF9T"
    ]
  },
  "kind": "statement",
  "link": "https://tamnd.github.io/math2/node/HK8339ECNN/",
  "links": [
    {
      "field": "about",
      "target": "X1D1C7VH6P"
    },
    {
      "field": "uses",
      "target": "0NFPJVJJHX"
    },
    {
      "field": "uses",
      "target": "556WBDAHNP"
    },
    {
      "field": "uses",
      "target": "AZRPHS478B"
    },
    {
      "field": "uses",
      "target": "X1D1C7VH6P"
    },
    {
      "field": "uses",
      "target": "RARTMA0ZEC"
    },
    {
      "field": "uses",
      "target": "08QMKTWF9T"
    }
  ],
  "markdown": "https://tamnd.github.io/math2/node/HK8339ECNN.md",
  "msc": [
    "37A35",
    "28D20",
    "94A17"
  ],
  "name": "Shannon–McMillan–Breiman theorem",
  "prov": null,
  "role": "theorem",
  "source": "https://github.com/tamnd/math2/blob/main/nodes/HK8/HK8339ECNN.md",
  "status": null,
  "tag": "HK8339ECNN",
  "uri": "m2://HK8339ECNN",
  "url": "https://tamnd.github.io/math2/wiki/shannon-mcmillan-breiman-theorem/",
  "writing": {
    "tag": "HK8339ECNN",
    "kind": "statement",
    "level": "W0",
    "next": [
      {
        "what": "an inventory row",
        "note": "the row of the inventory that the node accounts for",
        "have": 0,
        "want": 1
      },
      {
        "what": "a source with a locator",
        "have": 0,
        "want": 1
      }
    ]
  }
}