{
  "aliases": null,
  "backlinks": null,
  "body": "\n## Definition\n\nLet $X$ be a real $n \\times p$ matrix, let $y \\in \\R^n$, and let $\\lambda \u003e 0$.\nThe *ridge regression estimator* with parameter $\\lambda$ is the minimiser of\n$$\\norm{y - X b}^2 + \\lambda \\norm{b}^2$$\nover $b \\in \\R^p$, where $\\norm{\\cdot}$ is the Euclidean norm.\nIt equals\n$$\\hat\\beta_\\lambda = (X^T X + \\lambda I_p)^{-1} X^T y.$$\n\n## Remarks\n\nThe matrix $X^T X + \\lambda I_p$ is [[positive definite|0KX66CZMBB]] for $\\lambda \u003e 0$, so the minimiser exists and is unique for every $X$, whatever its rank.\nThe formula comes from setting the gradient $-2 X^T (y - X b) + 2 \\lambda b$ to zero.\n\nExample: take $p = 1$, $X = (1, 2)^T$, $y = (1, 2)^T$ and $\\lambda = 1$.\nThen $X^T X = 5$ and $X^T y = 5$, so $\\hat\\beta_1 = 5/6$, while the [[ordinary least squares|YM708KHV2W]] slope is $1$.\n\nNon-example: with $\\lambda = 0$ the problem is plain [[least squares|0BYTQ4X1HC]], and if $X^T X$ is singular, for example when two columns of $X$ are equal, the minimiser is not unique.\n\nIn practice the regressors are usually standardised and the intercept is not penalised.\nFor each $\\lambda \u003e 0$ there is $t \u003e 0$ such that $\\hat\\beta_\\lambda$ also minimises $\\norm{y - Xb}^2$ under the constraint $\\norm{b}^2 \\le t$.\nIn a normal linear model, $\\hat\\beta_\\lambda$ is the posterior mode for a centred normal prior on $\\beta$.\n",
  "commit": "ff8a40749dc16545e3d2fc637f6b0196e81d0bfa",
  "front": {
    "defines": "3KZW2930V2",
    "kind": "definition",
    "msc": [
      "62J07",
      "62J05"
    ],
    "name": "Ridge regression",
    "sources": [
      {
        "url": "https://en.wikipedia.org/wiki/Ridge_regression"
      }
    ],
    "tag": "KCVKGSYJR9",
    "uses": [
      "YM708KHV2W",
      "0BYTQ4X1HC",
      "0KX66CZMBB"
    ]
  },
  "kind": "definition",
  "link": "https://tamnd.github.io/math2/node/KCVKGSYJR9/",
  "links": [
    {
      "field": "defines",
      "target": "3KZW2930V2"
    },
    {
      "field": "uses",
      "target": "YM708KHV2W"
    },
    {
      "field": "uses",
      "target": "0BYTQ4X1HC"
    },
    {
      "field": "uses",
      "target": "0KX66CZMBB"
    }
  ],
  "markdown": "https://tamnd.github.io/math2/node/KCVKGSYJR9.md",
  "msc": [
    "62J07",
    "62J05"
  ],
  "name": "Ridge regression",
  "prov": null,
  "role": "",
  "source": "https://github.com/tamnd/math2/blob/main/nodes/KCV/KCVKGSYJR9.md",
  "status": null,
  "tag": "KCVKGSYJR9",
  "uri": "m2://KCVKGSYJR9",
  "url": "https://tamnd.github.io/math2/wiki/ridge-regression-definition/",
  "writing": {
    "tag": "KCVKGSYJR9",
    "kind": "definition",
    "level": "W0",
    "next": [
      {
        "what": "an inventory row",
        "note": "the row of the inventory that the node accounts for",
        "have": 0,
        "want": 1
      },
      {
        "what": "a source with a locator",
        "have": 0,
        "want": 1
      }
    ]
  }
}