{
  "aliases": null,
  "backlinks": null,
  "body": "\n## Definition\n\nLet $D$ be data from a finite or countable set, and let $\\mathcal{H}$ be a countable set of hypotheses.\nEach hypothesis $H \\in \\mathcal{H}$ is a [[probability distribution|0QB913WM5E]] $P_H$ on the possible data, as in a [[statistical model|K3AMMSY8DD]].\nFix a prefix code for $\\mathcal{H}$, with length $L(H)$ in bits for the codeword of $H$.\nLet $L(D \\mid H) = -\\log_2 P_H(D)$ be the code length of $D$ with the code built from $P_H$.\nThe *two-part minimum description length* principle selects a hypothesis $H \\in \\mathcal{H}$ that minimizes\n$$L(H) + L(D \\mid H).$$\n\n## Remarks\n\nThe length $-\\log_2 P_H(D)$ is the length, up to rounding, of a Shannon code for $D$ built from $P_H$.\nExample: for $n$ coin tosses with $h$ heads, take $\\mathcal{H} = \\set{\\text{fair coin}, \\text{coin with } p = h/n}$ with $L(H) = 1$ bit each.\nThe fair coin costs $1 + n$ bits; the other costs $1 + n\\, \\mathrm{H}(h/n)$ bits, where $\\mathrm{H}$ is the binary entropy, so this toy rule picks it whenever $h \\ne n/2$.\nThis is too generous, because the second hypothesis depends on the data.\nA real two-part code must also encode $h$, which costs about $\\log_2 (n + 1)$ bits, and then MDL picks the biased coin only when $h/n$ is far enough from $1/2$.\nNon-example: minimizing $L(D \\mid H)$ alone is maximum likelihood, not MDL, and it ignores the cost of complex hypotheses.\nRefined versions of MDL replace the two-part code by a single universal code, such as the normalized maximum likelihood code.\n",
  "commit": "ff8a40749dc16545e3d2fc637f6b0196e81d0bfa",
  "front": {
    "defines": "8JV9PPFRG4",
    "kind": "definition",
    "msc": [
      "62B10",
      "94A17",
      "68Q30"
    ],
    "name": "Minimum description length",
    "sources": [
      {
        "url": "https://en.wikipedia.org/wiki/Minimum_description_length"
      }
    ],
    "tag": "DWZYGTXKNS",
    "uses": [
      "K3AMMSY8DD",
      "0QB913WM5E"
    ]
  },
  "kind": "definition",
  "link": "https://tamnd.github.io/math2/node/DWZYGTXKNS/",
  "links": [
    {
      "field": "defines",
      "target": "8JV9PPFRG4"
    },
    {
      "field": "uses",
      "target": "K3AMMSY8DD"
    },
    {
      "field": "uses",
      "target": "0QB913WM5E"
    }
  ],
  "markdown": "https://tamnd.github.io/math2/node/DWZYGTXKNS.md",
  "msc": [
    "62B10",
    "94A17",
    "68Q30"
  ],
  "name": "Minimum description length",
  "prov": null,
  "role": "",
  "source": "https://github.com/tamnd/math2/blob/main/nodes/DWZ/DWZYGTXKNS.md",
  "status": null,
  "tag": "DWZYGTXKNS",
  "uri": "m2://DWZYGTXKNS",
  "url": "https://tamnd.github.io/math2/wiki/minimum-description-length-definition/",
  "writing": {
    "tag": "DWZYGTXKNS",
    "kind": "definition",
    "level": "W0",
    "next": [
      {
        "what": "an inventory row",
        "note": "the row of the inventory that the node accounts for",
        "have": 0,
        "want": 1
      },
      {
        "what": "a source with a locator",
        "have": 0,
        "want": 1
      }
    ]
  }
}