{
  "aliases": null,
  "backlinks": null,
  "body": "\n## Definition\n\nConsider the [[linear regression|41FVC4P5JB]] model $y = X \\beta + \\varepsilon$ with an $n \\times p$ design matrix $X$ and response $y \\in \\R^n$.\nThe *ordinary least squares* [[estimator|0SSS6ERQ0S]] of $\\beta$ is a vector $\\hat\\beta \\in \\R^p$ that minimises the residual sum of squares\n$$S(b) = \\abs{y - X b}^2 = \\sum_{i=1}^n (y_i - x_i^T b)^2, \\qquad b \\in \\R^p.$$\nIf $X$ has [[rank|0ZC9YTXZRY]] $p$, then $X^T X$ is [[invertible|0EB9N0S33C]] and the minimiser is unique:\n$$\\hat\\beta = (X^T X)^{-1} X^T y.$$\n\n## Remarks\n\nThe minimisers are exactly the solutions of the normal equations $X^T X b = X^T y$.\nThe fitted vector $\\hat y = X \\hat\\beta = H y$, with $H = X (X^T X)^{-1} X^T$, is the [[orthogonal projection|0B8KZRT28M]] of $y$ onto the column space of $X$.\nSo the residual vector $y - \\hat y$ is orthogonal to every column of $X$.\n\nExample: fit $y = b_0 + b_1 t$ to the points $(0, 0)$, $(1, 1)$, $(2, 3)$.\nHere $\\bar t = 1$ and $\\bar y = 4/3$, so $b_1 = \\sum (t_i - \\bar t)(y_i - \\bar y) / \\sum (t_i - \\bar t)^2 = 3/2$ and $b_0 = \\bar y - b_1 \\bar t = -1/6$.\nCheck: the residuals are $1/6$, $-1/3$, $1/6$, which sum to $0$ and are orthogonal to $(0, 1, 2)$.\nNon-example: if two columns of $X$ are equal, $X$ has rank less than $p$ and $X^T X$ is singular.\nThen $S$ has infinitely many minimisers, so the formula above does not apply and $\\beta$ is not identified.\n",
  "commit": "ff8a40749dc16545e3d2fc637f6b0196e81d0bfa",
  "front": {
    "defines": "YM708KHV2W",
    "kind": "definition",
    "msc": [
      "62J05"
    ],
    "name": "Ordinary least squares",
    "sources": [
      {
        "url": "https://en.wikipedia.org/wiki/Ordinary_least_squares"
      }
    ],
    "tag": "5Y3DYMPVEG",
    "uses": [
      "41FVC4P5JB",
      "0BYTQ4X1HC",
      "0ZC9YTXZRY",
      "0EB9N0S33C",
      "0B8KZRT28M",
      "0SSS6ERQ0S"
    ]
  },
  "kind": "definition",
  "link": "https://tamnd.github.io/math2/node/5Y3DYMPVEG/",
  "links": [
    {
      "field": "defines",
      "target": "YM708KHV2W"
    },
    {
      "field": "uses",
      "target": "41FVC4P5JB"
    },
    {
      "field": "uses",
      "target": "0BYTQ4X1HC"
    },
    {
      "field": "uses",
      "target": "0ZC9YTXZRY"
    },
    {
      "field": "uses",
      "target": "0EB9N0S33C"
    },
    {
      "field": "uses",
      "target": "0B8KZRT28M"
    },
    {
      "field": "uses",
      "target": "0SSS6ERQ0S"
    }
  ],
  "markdown": "https://tamnd.github.io/math2/node/5Y3DYMPVEG.md",
  "msc": [
    "62J05"
  ],
  "name": "Ordinary least squares",
  "prov": null,
  "role": "",
  "source": "https://github.com/tamnd/math2/blob/main/nodes/5Y3/5Y3DYMPVEG.md",
  "status": null,
  "tag": "5Y3DYMPVEG",
  "uri": "m2://5Y3DYMPVEG",
  "url": "https://tamnd.github.io/math2/wiki/ordinary-least-squares-definition/",
  "writing": {
    "tag": "5Y3DYMPVEG",
    "kind": "definition",
    "level": "W0",
    "next": [
      {
        "what": "an inventory row",
        "note": "the row of the inventory that the node accounts for",
        "have": 0,
        "want": 1
      },
      {
        "what": "a source with a locator",
        "have": 0,
        "want": 1
      }
    ]
  }
}