{
  "aliases": null,
  "backlinks": null,
  "body": "\n## Statement\n\nFor each integer $n \\ge 2$, let $\\Delta_n$ be the set of probability vectors $(p_1, \\dots, p_n)$ with $p_i \\ge 0$ and $p_1 + \\cdots + p_n = 1$.\nLet $H$ be a function defined on $\\bigcup_{n \\ge 2} \\Delta_n$ with real values.\nAssume:\n\n1. The function $p \\mapsto H(p, 1 - p)$ is [[continuous|0X3WGX95FE]] on $[0, 1]$.\n2. $H$ is symmetric: $H(p_1, \\dots, p_n)$ does not change when the $p_i$ are permuted.\n3. $H(\\frac{1}{2}, \\frac{1}{2}) = 1$.\n4. For all $n \\ge 2$, all $(p_1, \\dots, p_n) \\in \\Delta_n$ with $p_n \u003e 0$, and all $q_1, q_2 \\ge 0$ with $q_1 + q_2 = p_n$,\n$$H(p_1, \\dots, p_{n-1}, q_1, q_2) = H(p_1, \\dots, p_n) + p_n \\, H\\left(\\frac{q_1}{p_n}, \\frac{q_2}{p_n}\\right).$$\n\nThen for every $n \\ge 2$ and every $(p_1, \\dots, p_n) \\in \\Delta_n$,\n$$H(p_1, \\dots, p_n) = -\\sum_{i=1}^{n} p_i \\log_2 p_i,$$\nwith the convention $0 \\log_2 0 = 0$.\nConversely, this function has properties 1 to 4.\n\n## Remarks\n\nThe formula is the Shannon entropy of a finite [[probability distribution|0DVDCMA2DK]], measured in bits.\nThis set of axioms is due to Faddeev; Shannon and Khinchin gave earlier characterizations with stronger hypotheses.\nThe proof first studies $F(n) = H(\\frac{1}{n}, \\dots, \\frac{1}{n})$: property 4 gives $F(mn) = F(m) + F(n)$, and the remaining properties force $F(n) = \\log_2 n$.\nGrouping equal masses then gives the formula for rational probabilities, and continuity extends it to all probability vectors.\nThe normalization 3 only fixes the unit; without it, $H$ is a constant multiple $c$ of the Shannon entropy.\n\n## Explanations\n\n### L1 human\n\nEntropy measures how uncertain the outcome of a random experiment is.\nThis theorem says that once one asks for a few natural rules, there is only one possible formula.\nThe first rule says that small changes of the probabilities give small changes of the uncertainty.\nThe other rules say that the order of the outcomes does not matter, that a fair coin carries one bit, and that uncertainty can be computed in stages.\nThe last rule says that splitting one outcome into two adds the uncertainty of the split, weighted by the probability of that outcome.\n\n### L2 human\n\nThe theorem is a solution of a [[functional equation|SRAN4EKWS0]]: setting $f(p) = H(p, 1 - p)$, the properties reduce to the fundamental equation of information $f(x) + (1 - x) f\\left(\\frac{y}{1 - x}\\right) = f(y) + (1 - y) f\\left(\\frac{x}{1 - y}\\right)$ for $x, y \\in [0, 1)$ with $x + y \\le 1$.\nIt explains why the logarithm, and no other function, appears in the entropy formula.\nDropping or changing the recursivity axiom leads to other entropies, such as the Rényi entropies.\nSimilar axiomatic characterizations exist for relative entropy and mutual information.\n",
  "commit": "ff8a40749dc16545e3d2fc637f6b0196e81d0bfa",
  "front": {
    "about": [
      "0DVDCMA2DK"
    ],
    "kind": "statement",
    "msc": [
      "94A17",
      "39B22"
    ],
    "name": "Characterization of Shannon entropy",
    "role": "theorem",
    "sources": [
      {
        "url": "https://en.wikipedia.org/wiki/Entropy_(information_theory)"
      }
    ],
    "tag": "3TERJ5C8QK",
    "uses": [
      "0DVDCMA2DK",
      "0X3WGX95FE",
      "SRAN4EKWS0"
    ]
  },
  "kind": "statement",
  "link": "https://tamnd.github.io/math2/node/3TERJ5C8QK/",
  "links": [
    {
      "field": "about",
      "target": "0DVDCMA2DK"
    },
    {
      "field": "uses",
      "target": "0DVDCMA2DK"
    },
    {
      "field": "uses",
      "target": "0X3WGX95FE"
    },
    {
      "field": "uses",
      "target": "SRAN4EKWS0"
    }
  ],
  "markdown": "https://tamnd.github.io/math2/node/3TERJ5C8QK.md",
  "msc": [
    "94A17",
    "39B22"
  ],
  "name": "Characterization of Shannon entropy",
  "prov": null,
  "role": "theorem",
  "source": "https://github.com/tamnd/math2/blob/main/nodes/3TE/3TERJ5C8QK.md",
  "status": null,
  "tag": "3TERJ5C8QK",
  "uri": "m2://3TERJ5C8QK",
  "url": "https://tamnd.github.io/math2/wiki/characterization-of-shannon-entropy/",
  "writing": {
    "tag": "3TERJ5C8QK",
    "kind": "statement",
    "level": "W0",
    "next": [
      {
        "what": "an inventory row",
        "note": "the row of the inventory that the node accounts for",
        "have": 0,
        "want": 1
      },
      {
        "what": "a source with a locator",
        "have": 0,
        "want": 1
      }
    ]
  }
}