{
  "aliases": null,
  "backlinks": [
    {
      "field": "uses",
      "target": "8DWRYNAMFT"
    }
  ],
  "body": "\n## Statement\n\nLet $\\set{f(\\cdot; \\theta) : \\theta \\in \\Theta}$ be a family of probability densities with respect to a measure $\\nu$, and let $X_1, X_2, \\dots$ be [[i.i.d.|0AJN7WHBJZ]] with density $f(\\cdot; \\theta_0)$ for some $\\theta_0 \\in \\Theta$.\nAssume:\n\n1. ([[identifiability|605M5NMFD5]]) if $\\theta \\ne \\theta_0$, then $f(\\cdot; \\theta) \\ne f(\\cdot; \\theta_0)$ on a set of positive $\\nu$-measure;\n2. $\\Theta$ is a [[compact|00R0X1WGPX]] subset of $\\R^k$;\n3. for $\\nu$-almost every $x$, the map $\\theta \\mapsto \\log f(x; \\theta)$ is continuous on $\\Theta$;\n4. $E_{\\theta_0}\\big[\\sup_{\\theta \\in \\Theta} \\abs{\\log f(X_1; \\theta)}\\big] \u003c \\infty$.\n\nLet $\\hat{\\theta}_n$ be a measurable maximizer over $\\Theta$ of the log-[[likelihood|05FSTXM92W]] $\\ell_n(\\theta) = \\sum_{i=1}^n \\log f(X_i; \\theta)$.\nThen $\\hat{\\theta}_n \\to \\theta_0$ in [[probability|0EFDDXTVD6]]: the [[maximum likelihood estimator|0FREPFKNS7]] is [[consistent|0XJVF81QF6]].\n\n## Remarks\n\nIdea of proof: let $L(\\theta) = E_{\\theta_0}[\\log f(X_1; \\theta)]$.\nBy [[Jensen's inequality|0RWMRTPC87]] for the strictly concave logarithm, and by identifiability, $L(\\theta) \u003c L(\\theta_0)$ for every $\\theta \\ne \\theta_0$.\nConditions 2 to 4 give a uniform [[law of large numbers|0BFDHF2H7N]]: $\\sup_{\\theta} \\abs{\\ell_n(\\theta)/n - L(\\theta)} \\to 0$ in probability.\nSince $L$ is continuous on the compact set $\\Theta$, it stays below $L(\\theta_0)$ by a fixed margin outside each neighbourhood of $\\theta_0$.\nSo the maximizer of $\\ell_n/n$ must eventually lie inside that neighbourhood.\n\nExample: for i.i.d. $\\mathcal{N}(\\mu, 1)$ data with $\\mu \\in [-K, K]$, the conditions hold and $\\hat{\\mu}_n$ is the sample mean clipped to $[-K, K]$, which tends to $\\mu_0$.\n\nIdentifiability is necessary: if $f(\\cdot; \\theta_1) = f(\\cdot; \\theta_0)$ for some $\\theta_1 \\ne \\theta_0$, the data cannot distinguish $\\theta_0$ from $\\theta_1$, and no estimator converges to $\\theta_0$ under both.\nCompactness can be replaced by other conditions, such as concavity of $\\theta \\mapsto \\log f(x; \\theta)$, but some condition of this kind is needed.\n\n## Explanations\n\n### L1 human\n\nWith more and more data, the parameter value that makes the data most likely gets closer and closer to the true value.\nThe reason is that, on average, the true parameter gives the data a higher log-likelihood than any other parameter.\nThe conditions make sure that this average behaviour is reached uniformly over all parameter values.\n\n### L2 human\n\nThis is Wald's approach to consistency, and the same argument proves consistency of general M-estimators and extremum estimators.\nConsistency is the first step in proving the asymptotic normality of the maximum likelihood estimator, which needs a consistent estimator to linearize the likelihood equations around $\\theta_0$.\nThe limit $L(\\theta_0) - L(\\theta)$ is the Kullback–Leibler divergence of $f(\\cdot; \\theta)$ from $f(\\cdot; \\theta_0)$, so maximum likelihood can be seen as minimizing an empirical version of this divergence.\n",
  "commit": "ff8a40749dc16545e3d2fc637f6b0196e81d0bfa",
  "front": {
    "about": [
      "0FREPFKNS7",
      "0XJVF81QF6"
    ],
    "kind": "statement",
    "msc": [
      "62F12",
      "62F10"
    ],
    "name": "Consistency of the maximum likelihood estimator",
    "role": "theorem",
    "sources": [
      {
        "locator": "Consistency",
        "url": "https://en.wikipedia.org/wiki/Maximum_likelihood_estimation"
      }
    ],
    "tag": "54387XJ9QW",
    "uses": [
      "0FREPFKNS7",
      "05FSTXM92W",
      "0XJVF81QF6",
      "0AJN7WHBJZ",
      "00R0X1WGPX",
      "0EFDDXTVD6",
      "605M5NMFD5",
      "0RWMRTPC87",
      "0BFDHF2H7N"
    ]
  },
  "kind": "statement",
  "link": "https://tamnd.github.io/math2/node/54387XJ9QW/",
  "links": [
    {
      "field": "about",
      "target": "0FREPFKNS7"
    },
    {
      "field": "about",
      "target": "0XJVF81QF6"
    },
    {
      "field": "uses",
      "target": "0FREPFKNS7"
    },
    {
      "field": "uses",
      "target": "05FSTXM92W"
    },
    {
      "field": "uses",
      "target": "0XJVF81QF6"
    },
    {
      "field": "uses",
      "target": "0AJN7WHBJZ"
    },
    {
      "field": "uses",
      "target": "00R0X1WGPX"
    },
    {
      "field": "uses",
      "target": "0EFDDXTVD6"
    },
    {
      "field": "uses",
      "target": "605M5NMFD5"
    },
    {
      "field": "uses",
      "target": "0RWMRTPC87"
    },
    {
      "field": "uses",
      "target": "0BFDHF2H7N"
    }
  ],
  "markdown": "https://tamnd.github.io/math2/node/54387XJ9QW.md",
  "msc": [
    "62F12",
    "62F10"
  ],
  "name": "Consistency of the maximum likelihood estimator",
  "prov": null,
  "role": "theorem",
  "source": "https://github.com/tamnd/math2/blob/main/nodes/543/54387XJ9QW.md",
  "status": null,
  "tag": "54387XJ9QW",
  "uri": "m2://54387XJ9QW",
  "url": "https://tamnd.github.io/math2/wiki/consistency-of-the-maximum-likelihood-estimator/",
  "writing": {
    "tag": "54387XJ9QW",
    "kind": "statement",
    "level": "W0",
    "next": [
      {
        "what": "an inventory row",
        "note": "the row of the inventory that the node accounts for",
        "have": 0,
        "want": 1
      }
    ]
  }
}