{
  "aliases": null,
  "backlinks": null,
  "body": "\n## Statement\n\nLet $x$ be a fixed point and let $Y = f(x) + \\varepsilon$, where $f(x)$ is a real number and $\\varepsilon$ is a random variable with $E(\\varepsilon) = 0$ and $\\operatorname{Var}(\\varepsilon) = \\sigma^2$.\nLet $\\hat{f}(x)$ be a random variable with $E(\\hat{f}(x)^2) \u003c \\infty$ that is [[independent|0E9502NT92]] of $\\varepsilon$.\nFor example, $\\hat{f}(x)$ is a prediction built from a training sample independent of the new observation.\nThen\n$$E\\big[(Y - \\hat{f}(x))^2\\big] = \\big(E[\\hat{f}(x)] - f(x)\\big)^2 + \\operatorname{Var}\\big(\\hat{f}(x)\\big) + \\sigma^2.$$\n\n## Remarks\n\nProof: write $Y - \\hat{f} = \\varepsilon + (f - E\\hat{f}) + (E\\hat{f} - \\hat{f})$.\nExpand the square and take [[expectations|0DCT0QZVQ1]]: the three cross terms vanish, because $E\\varepsilon = 0$, $E(E\\hat{f} - \\hat{f}) = 0$, and $\\varepsilon$ is independent of $\\hat{f}$.\nThe remaining terms are $\\sigma^2$, the squared bias and the [[variance|007QED3E0Y]] of $\\hat{f}$.\n\nThe case without noise is the decomposition of the [[mean squared error|0V4VSCBXP3]] of an estimator $\\hat{\\theta}$ of a parameter $\\theta$:\n$$E\\big[(\\hat{\\theta} - \\theta)^2\\big] = \\operatorname{Bias}(\\hat{\\theta})^2 + \\operatorname{Var}(\\hat{\\theta}),$$\nwith the [[bias|01BD2TFFP4]] $\\operatorname{Bias}(\\hat{\\theta}) = E(\\hat{\\theta}) - \\theta$.\n\nExample: for i.i.d. $X_1, \\dots, X_n$ with variance $\\sigma^2$, the estimator $\\hat{\\sigma}^2 = \\frac{1}{n}\\sum_i (X_i - \\bar{X})^2$ has bias $-\\sigma^2/n$.\nFor normal data its variance is $2(n-1)\\sigma^4/n^2$, so its mean squared error is $(2n - 1)\\sigma^4/n^2$, which is smaller than the mean squared error $2\\sigma^4/(n-1)$ of the unbiased sample variance.\nA small bias can thus buy a larger reduction of variance.\n\nThe irreducible term $\\sigma^2$ does not depend on $\\hat{f}$; no predictor can do better than it on average.\n\n## Explanations\n\n### L1 human\n\nThe average squared error of a prediction has three parts.\nThe bias is how far the prediction is off on average.\nThe variance is how much the prediction jumps around when the training data change.\nThe noise is the part of the outcome that nobody can predict.\nSimple models tend to have large bias and small variance, flexible models the opposite.\n\n### L2 human\n\nThe decomposition is the basis of the bias–variance tradeoff in model selection: adding complexity lowers the bias but raises the variance, and the best expected error lies in between.\nShrinkage methods such as ridge regression accept bias to reduce variance.\nIn point estimation it explains why an unbiased estimator need not have the smallest mean squared error.\nNote that the decomposition holds at each fixed point $x$; averaging over $x$ gives the decomposition of the expected prediction error.\n",
  "commit": "ff8a40749dc16545e3d2fc637f6b0196e81d0bfa",
  "front": {
    "about": [
      "05NZ1HHQMV"
    ],
    "kind": "statement",
    "msc": [
      "62F10",
      "62J05"
    ],
    "name": "Bias–variance decomposition",
    "role": "proposition",
    "sources": [
      {
        "url": "https://en.wikipedia.org/wiki/Bias–variance_tradeoff"
      }
    ],
    "tag": "CJQ8D9FTQN",
    "uses": [
      "0V4VSCBXP3",
      "01BD2TFFP4",
      "007QED3E0Y",
      "0DCT0QZVQ1",
      "0E9502NT92"
    ]
  },
  "kind": "statement",
  "link": "https://tamnd.github.io/math2/node/CJQ8D9FTQN/",
  "links": [
    {
      "field": "about",
      "target": "05NZ1HHQMV"
    },
    {
      "field": "uses",
      "target": "0V4VSCBXP3"
    },
    {
      "field": "uses",
      "target": "01BD2TFFP4"
    },
    {
      "field": "uses",
      "target": "007QED3E0Y"
    },
    {
      "field": "uses",
      "target": "0DCT0QZVQ1"
    },
    {
      "field": "uses",
      "target": "0E9502NT92"
    }
  ],
  "markdown": "https://tamnd.github.io/math2/node/CJQ8D9FTQN.md",
  "msc": [
    "62F10",
    "62J05"
  ],
  "name": "Bias–variance decomposition",
  "prov": null,
  "role": "proposition",
  "source": "https://github.com/tamnd/math2/blob/main/nodes/CJQ/CJQ8D9FTQN.md",
  "status": null,
  "tag": "CJQ8D9FTQN",
  "uri": "m2://CJQ8D9FTQN",
  "url": "https://tamnd.github.io/math2/wiki/bias-variance-decomposition/",
  "writing": {
    "tag": "CJQ8D9FTQN",
    "kind": "statement",
    "level": "W0",
    "next": [
      {
        "what": "an inventory row",
        "note": "the row of the inventory that the node accounts for",
        "have": 0,
        "want": 1
      },
      {
        "what": "a source with a locator",
        "have": 0,
        "want": 1
      }
    ]
  }
}