{
  "aliases": null,
  "backlinks": [
    {
      "field": "uses",
      "target": "XK7W504YEW"
    }
  ],
  "body": "\n## Statement\n\nLet $X$ be a fixed real $n \\times p$ matrix of rank $p$, and let $\\beta \\in \\R^p$ be an unknown parameter.\nLet $\\varepsilon$ be a random vector in $\\R^n$ and\n$$y = X\\beta + \\varepsilon.$$\nAssume the following.\n\n1. $E[\\varepsilon] = 0$.\n2. $\\operatorname{Cov}(\\varepsilon) = \\sigma^2 I_n$ for some unknown $\\sigma^2 \u003e 0$, so the errors are uncorrelated and [[homoscedastic|D8VX6XQ3TT]].\n\nLet $\\hat\\beta = (X^T X)^{-1} X^T y$ be the [[ordinary least squares|YM708KHV2W]] estimator.\nLet $\\tilde\\beta = C y$ be any estimator with $C$ a fixed $p \\times n$ matrix such that $E[\\tilde\\beta] = \\beta$ for every $\\beta \\in \\R^p$.\nThen $\\hat\\beta$ is [[unbiased|0BWGTN92FG]], and the matrix $\\operatorname{Cov}(\\tilde\\beta) - \\operatorname{Cov}(\\hat\\beta)$ is [[positive semidefinite|01MMVW8CKC]].\nIn particular, $\\operatorname{Var}(c^T \\tilde\\beta) \\ge \\operatorname{Var}(c^T \\hat\\beta)$ for every $c \\in \\R^p$.\n\n## Remarks\n\nWrite $C = (X^T X)^{-1} X^T + D$.\nUnbiasedness for every $\\beta$ forces $D X = 0$.\nThen $\\operatorname{Cov}(\\tilde\\beta) = \\sigma^2 C C^T = \\sigma^2 (X^T X)^{-1} + \\sigma^2 D D^T$, because the cross terms contain $D X$.\nThe matrix $D D^T$ is positive semidefinite, which gives the claim.\n\nFor the model $y_i = \\mu + \\varepsilon_i$, the estimator $\\hat\\beta$ is the sample mean.\nHere the theorem says that the sample mean has the smallest variance among all weighted averages $\\sum_i c_i y_i$ with $\\sum_i c_i = 1$.\n\nThe errors need not be normal or independent.\nIf the [[covariance matrix|0V7G6WMFK0]] of $\\varepsilon$ is not a multiple of $I_n$, then ordinary least squares is still unbiased but in general no longer best.\nThe theorem compares only linear unbiased estimators: a biased estimator, such as a ridge estimator, can have a smaller mean squared error.\n\n## Explanations\n\n### L1 human\n\nIn a linear model with well-behaved noise, the least squares fit is the most precise estimator among those that are linear in the data and correct on average.\n\"Best linear unbiased estimator\", or BLUE, is the short name for this property.\nIt is the main reason why least squares is the default method for fitting linear models.\n\n### L2 human\n\nThe hypotheses are only about the first two moments of the errors: mean zero, equal variances and no correlation.\nWhen the errors are also normal, a stronger result holds: least squares is then the minimum-variance unbiased estimator among all unbiased estimators, linear or not.\nWhen the variances differ or the errors are correlated with a known covariance structure, one transforms the model to restore the hypotheses.\nThat leads to generalized and weighted least squares.\nThe theorem says nothing about biased estimators, and shrinkage methods trade a small bias for a large drop in variance.\n",
  "commit": "ff8a40749dc16545e3d2fc637f6b0196e81d0bfa",
  "front": {
    "about": [
      "YM708KHV2W",
      "41FVC4P5JB"
    ],
    "kind": "statement",
    "msc": [
      "62J05"
    ],
    "name": "Gauss–Markov theorem",
    "role": "theorem",
    "same_as": {
      "wikidata": "Q428134",
      "wikipedia": "Gauss–Markov theorem"
    },
    "sources": [
      {
        "url": "https://en.wikipedia.org/wiki/Gauss–Markov_theorem"
      }
    ],
    "tag": "N3YMNMWANM",
    "uses": [
      "A06SGFCSCE",
      "YM708KHV2W",
      "0V7G6WMFK0",
      "0BWGTN92FG",
      "0477R9PYAE",
      "01MMVW8CKC",
      "D8VX6XQ3TT"
    ]
  },
  "kind": "statement",
  "link": "https://tamnd.github.io/math2/node/N3YMNMWANM/",
  "links": [
    {
      "field": "about",
      "target": "YM708KHV2W"
    },
    {
      "field": "about",
      "target": "41FVC4P5JB"
    },
    {
      "field": "uses",
      "target": "A06SGFCSCE"
    },
    {
      "field": "uses",
      "target": "YM708KHV2W"
    },
    {
      "field": "uses",
      "target": "0V7G6WMFK0"
    },
    {
      "field": "uses",
      "target": "0BWGTN92FG"
    },
    {
      "field": "uses",
      "target": "0477R9PYAE"
    },
    {
      "field": "uses",
      "target": "01MMVW8CKC"
    },
    {
      "field": "uses",
      "target": "D8VX6XQ3TT"
    }
  ],
  "markdown": "https://tamnd.github.io/math2/node/N3YMNMWANM.md",
  "msc": [
    "62J05"
  ],
  "name": "Gauss–Markov theorem",
  "prov": null,
  "role": "theorem",
  "source": "https://github.com/tamnd/math2/blob/main/nodes/N3Y/N3YMNMWANM.md",
  "status": null,
  "tag": "N3YMNMWANM",
  "uri": "m2://N3YMNMWANM",
  "url": "https://tamnd.github.io/math2/wiki/gauss-markov-theorem/",
  "writing": {
    "tag": "N3YMNMWANM",
    "kind": "statement",
    "level": "W0",
    "next": [
      {
        "what": "an inventory row",
        "note": "the row of the inventory that the node accounts for",
        "have": 0,
        "want": 1
      },
      {
        "what": "a source with a locator",
        "have": 0,
        "want": 1
      }
    ]
  }
}