{
  "aliases": null,
  "backlinks": null,
  "body": "\n## Definition\n\nLet $A \\in \\R^{m \\times n}$ and $b \\in \\R^m$.\nThe *linear least squares problem* is to find $x \\in \\R^n$ that minimizes $\\norm{Ax - b}_2$.\nWhen $A$ has rank less than $n$, one asks for the minimizer of smallest norm.\nThis is $x = A^\\dagger b$, with $A^\\dagger$ the [[Moore–Penrose inverse|0YY8VX28JR]].\nA *numerical method for linear least squares* is an algorithm that computes this minimizer from $A$ and $b$.\nIt is exact in exact arithmetic, and accurate up to rounding errors in floating point arithmetic.\nThe standard methods are:\n\n1. the normal equations $A^T A x = A^T b$, solved by a [[Cholesky decomposition|0WKG8CYY6A]] of $A^T A$ when $A$ has full column rank;\n2. a [[QR decomposition|0ERWG4NXQ9]] $A = QR$ with $Q \\in \\R^{m \\times n}$ having orthonormal columns and $R$ upper triangular, when $A$ has full column rank, followed by the solution of $R x = Q^T b$;\n3. the singular value decomposition $A = U \\Sigma V^T$, followed by $x = V \\Sigma^\\dagger U^T b$.\n\n## Remarks\n\nExample: for fitting a line $y = c_0 + c_1 t$ to the points $(0, 0)$, $(1, 1)$, $(2, 1)$, take $A$ with rows $(1, 0)$, $(1, 1)$, $(1, 2)$ and $b = (0, 1, 1)$.\nThen $A^T A = \\begin{pmatrix} 3 \u0026 3 \\\\ 3 \u0026 5 \\end{pmatrix}$ and $A^T b = (2, 3)$, so $c_0 = 1/6$ and $c_1 = 1/2$; the residual $(-1/6, 1/3, -1/6)$ is orthogonal to both columns of $A$, as it must be.\nNon-example: solving the square system formed by the first $n$ rows of $Ax = b$ fits those rows exactly, but it does not give the minimizer in general.\nHere it gives $c_0 = 0$, $c_1 = 1$, with residual norm $1$ instead of $1/\\sqrt{6}$.\nThe normal equations are the cheapest method but square the condition number, $\\kappa(A^T A) = \\kappa(A)^2$.\nHouseholder QR is backward stable and is the standard choice for dense full-rank problems; the SVD is the most robust for rank-deficient problems, at a higher cost.\nFor large sparse problems one uses iterative methods such as LSQR or the conjugate gradient method applied to the normal equations.\n\n## Explanations\n\n### L1 human\n\nAll these methods compute the same answer in exact arithmetic: the best fit in the sense of squared errors.\nThey differ in speed and in how many digits they lose to rounding.\n",
  "commit": "ff8a40749dc16545e3d2fc637f6b0196e81d0bfa",
  "front": {
    "defines": "AWQA2Y30N4",
    "kind": "definition",
    "msc": [
      "65F20",
      "62J05"
    ],
    "name": "Numerical methods for linear least squares",
    "sources": [
      {
        "url": "https://en.wikipedia.org/wiki/Numerical_methods_for_linear_least_squares"
      }
    ],
    "tag": "V9R8K4JCVM",
    "uses": [
      "0BYTQ4X1HC",
      "0YY8VX28JR",
      "0ERWG4NXQ9",
      "0WKG8CYY6A"
    ]
  },
  "kind": "definition",
  "link": "https://tamnd.github.io/math2/node/V9R8K4JCVM/",
  "links": [
    {
      "field": "defines",
      "target": "AWQA2Y30N4"
    },
    {
      "field": "uses",
      "target": "0BYTQ4X1HC"
    },
    {
      "field": "uses",
      "target": "0YY8VX28JR"
    },
    {
      "field": "uses",
      "target": "0ERWG4NXQ9"
    },
    {
      "field": "uses",
      "target": "0WKG8CYY6A"
    }
  ],
  "markdown": "https://tamnd.github.io/math2/node/V9R8K4JCVM.md",
  "msc": [
    "65F20",
    "62J05"
  ],
  "name": "Numerical methods for linear least squares",
  "prov": null,
  "role": "",
  "source": "https://github.com/tamnd/math2/blob/main/nodes/V9R/V9R8K4JCVM.md",
  "status": null,
  "tag": "V9R8K4JCVM",
  "uri": "m2://V9R8K4JCVM",
  "url": "https://tamnd.github.io/math2/wiki/numerical-methods-for-linear-least-squares-definition/",
  "writing": {
    "tag": "V9R8K4JCVM",
    "kind": "definition",
    "level": "W0",
    "next": [
      {
        "what": "an inventory row",
        "note": "the row of the inventory that the node accounts for",
        "have": 0,
        "want": 1
      },
      {
        "what": "a source with a locator",
        "have": 0,
        "want": 1
      }
    ]
  }
}