{
  "aliases": null,
  "backlinks": null,
  "body": "\n## Statement\n\nLet $\\set{f(\\cdot; \\theta) : \\theta \\in \\Theta}$ be a family of probability densities with respect to a measure $\\nu$, with $\\Theta \\subseteq \\R^k$, and let $X_1, X_2, \\dots$ be [[i.i.d.|0AJN7WHBJZ]] with density $f(\\cdot; \\theta_0)$.\nAssume:\n\n1. $\\theta_0$ lies in the [[interior|0C38S0KMJK]] of $\\Theta$;\n2. on a neighbourhood $N$ of $\\theta_0$, $f(x; \\theta) \u003e 0$ and $\\theta \\mapsto f(x; \\theta)$ is twice continuously differentiable, for every $x$;\n3. $\\int \\sup_{\\theta \\in N} \\norm{\\nabla_\\theta f(x; \\theta)} \\, d\\nu(x) \u003c \\infty$ and $\\int \\sup_{\\theta \\in N} \\norm{\\nabla^2_\\theta f(x; \\theta)} \\, d\\nu(x) \u003c \\infty$;\n4. the [[Fisher information|G1Z3WSTE8Q]] $I(\\theta_0) = E_{\\theta_0}\\big[s(\\theta_0; X_1)\\, s(\\theta_0; X_1)^T\\big]$ exists and is nonsingular, where $s(\\theta; x) = \\nabla_\\theta \\log f(x; \\theta)$ is the [[score|CP97GRVTQ0]];\n5. $E_{\\theta_0}\\big[\\sup_{\\theta \\in N} \\norm{\\nabla^2_\\theta \\log f(X_1; \\theta)}\\big] \u003c \\infty$.\n\nLet $\\hat{\\theta}_n$ be a [[maximum likelihood estimator|0FREPFKNS7]] with $\\hat{\\theta}_n \\to \\theta_0$ in probability.\nThen\n$$\\sqrt{n}\\,(\\hat{\\theta}_n - \\theta_0) \\xrightarrow{d} \\mathcal{N}_k\\big(0, I(\\theta_0)^{-1}\\big),$$\nwhere $\\xrightarrow{d}$ is [[convergence in distribution|0JDAQHWA9N]] and $\\mathcal{N}_k$ is the [[multivariate normal distribution|0G558GJQMW]].\n\n## Remarks\n\nIdea of proof: with probability tending to one, $\\hat{\\theta}_n$ is an interior maximizer, so the score equation $\\sum_i s(\\hat{\\theta}_n; X_i) = 0$ holds.\nA Taylor expansion around $\\theta_0$ gives $0 = \\sum_i s(\\theta_0; X_i) + H_n (\\hat{\\theta}_n - \\theta_0)$, where $H_n$ is the [[Hessian|0693Y2METT]] of the log-likelihood at intermediate points.\nConditions 2 and 3 allow differentiation under the integral, so the score has mean $0$ and covariance $I(\\theta_0)$, and the [[central limit theorem|0EPZC4Z1FC]] gives $n^{-1/2}\\sum_i s(\\theta_0; X_i) \\xrightarrow{d} \\mathcal{N}_k(0, I(\\theta_0))$.\nCondition 5 and consistency give $H_n/n \\to -I(\\theta_0)$ in probability, and solving for $\\sqrt{n}(\\hat{\\theta}_n - \\theta_0)$ gives the result.\nConsistency itself is provided, for example, by the [[consistency theorem|54387XJ9QW]].\n\nExample: for i.i.d. Exponential($\\lambda$) data, $\\hat{\\lambda}_n = 1/\\bar{X}_n$ and $I(\\lambda) = 1/\\lambda^2$, so $\\sqrt{n}(\\hat{\\lambda}_n - \\lambda) \\xrightarrow{d} \\mathcal{N}(0, \\lambda^2)$.\n\nThe interior condition is needed.\nFor $\\mathcal{N}(\\mu, 1)$ data with $\\Theta = [0, \\infty)$ and $\\mu_0 = 0$, the MLE is $\\max(\\bar{X}_n, 0)$.\nThen $\\sqrt{n}\\,\\hat{\\mu}_n$ converges to the law of $\\max(Z, 0)$ with $Z$ standard normal, which is not normal.\nSmoothness in $\\theta$ with a fixed support is also needed: for Uniform$(0, \\theta)$ data the MLE is $\\max_i X_i$, and $n(\\theta_0 - \\hat{\\theta}_n)$ has an exponential limit.\n\n## Explanations\n\n### L1 human\n\nFor large samples, the maximum likelihood estimate is approximately normally distributed around the true value.\nIts spread shrinks like $1/\\sqrt{n}$, and the constant is the inverse of the Fisher information, which measures how sharply the likelihood peaks.\nThis is the smallest spread allowed by the Cramér–Rao bound, so maximum likelihood is asymptotically efficient.\n\n### L2 human\n\nThe theorem justifies the usual Wald confidence intervals $\\hat{\\theta}_n \\pm 1.96\\, (n\\, I(\\hat{\\theta}_n))^{-1/2}$ and the Wald test.\nTogether with the delta method it gives limit laws for smooth functions of the MLE.\nThe same expansion of the log-likelihood leads to Wilks' theorem for the likelihood-ratio statistic and to the score test.\nThe modern view puts the result in the frame of local asymptotic normality, where the bound $I(\\theta_0)^{-1}$ appears in the Hájek–Le Cam convolution theorem.\n",
  "commit": "ff8a40749dc16545e3d2fc637f6b0196e81d0bfa",
  "front": {
    "about": [
      "0FREPFKNS7"
    ],
    "kind": "statement",
    "msc": [
      "62F12",
      "62F10"
    ],
    "name": "Asymptotic normality of the maximum likelihood estimator",
    "role": "theorem",
    "sources": [
      {
        "url": "https://en.wikipedia.org/wiki/Maximum_likelihood_estimation"
      }
    ],
    "tag": "8DWRYNAMFT",
    "uses": [
      "0FREPFKNS7",
      "54387XJ9QW",
      "CP97GRVTQ0",
      "G1Z3WSTE8Q",
      "0AJN7WHBJZ",
      "0JDAQHWA9N",
      "0G558GJQMW",
      "0C38S0KMJK",
      "0693Y2METT",
      "0EPZC4Z1FC"
    ]
  },
  "kind": "statement",
  "link": "https://tamnd.github.io/math2/node/8DWRYNAMFT/",
  "links": [
    {
      "field": "about",
      "target": "0FREPFKNS7"
    },
    {
      "field": "uses",
      "target": "0FREPFKNS7"
    },
    {
      "field": "uses",
      "target": "54387XJ9QW"
    },
    {
      "field": "uses",
      "target": "CP97GRVTQ0"
    },
    {
      "field": "uses",
      "target": "G1Z3WSTE8Q"
    },
    {
      "field": "uses",
      "target": "0AJN7WHBJZ"
    },
    {
      "field": "uses",
      "target": "0JDAQHWA9N"
    },
    {
      "field": "uses",
      "target": "0G558GJQMW"
    },
    {
      "field": "uses",
      "target": "0C38S0KMJK"
    },
    {
      "field": "uses",
      "target": "0693Y2METT"
    },
    {
      "field": "uses",
      "target": "0EPZC4Z1FC"
    }
  ],
  "markdown": "https://tamnd.github.io/math2/node/8DWRYNAMFT.md",
  "msc": [
    "62F12",
    "62F10"
  ],
  "name": "Asymptotic normality of the maximum likelihood estimator",
  "prov": null,
  "role": "theorem",
  "source": "https://github.com/tamnd/math2/blob/main/nodes/8DW/8DWRYNAMFT.md",
  "status": null,
  "tag": "8DWRYNAMFT",
  "uri": "m2://8DWRYNAMFT",
  "url": "https://tamnd.github.io/math2/wiki/asymptotic-normality-of-the-maximum-likelihood-estimator/",
  "writing": {
    "tag": "8DWRYNAMFT",
    "kind": "statement",
    "level": "W0",
    "next": [
      {
        "what": "an inventory row",
        "note": "the row of the inventory that the node accounts for",
        "have": 0,
        "want": 1
      },
      {
        "what": "a source with a locator",
        "have": 0,
        "want": 1
      }
    ]
  }
}