{
  "aliases": null,
  "backlinks": [
    {
      "field": "uses",
      "target": "TH54SPVN1N"
    }
  ],
  "body": "\n## Statement\n\nLet $D \\subseteq \\R^n$ be open, let $m \\ge n$, and let $r \\colon D \\to \\R^m$ be continuously differentiable with [[Jacobian matrix|047NDTEPCV]] $J(x)$.\nThe aim is to minimize $S(x) = \\tfrac12 \\norm{r(x)}_2^2$.\nThe *Gauss–Newton algorithm* starts from $x_0 \\in D$ and sets\n$$x_{k+1} = x_k - \\bigl( J(x_k)^T J(x_k) \\bigr)^{-1} J(x_k)^T r(x_k),$$\nas long as $J(x_k)$ has full column rank.\nSuppose that $x^* \\in D$ satisfies $r(x^*) = 0$, that $J(x^*)$ has full column rank, and that $J$ is [[Lipschitz continuous|0QNVQ4VD9T]] near $x^*$.\nThen there are $\\delta \u003e 0$ and $C \u003e 0$ such that, for every $x_0$ with $\\norm{x_0 - x^*} \u003c \\delta$, the iterates are defined, converge to $x^*$, and satisfy\n$$\\norm{x_{k+1} - x^*} \\le C \\norm{x_k - x^*}^2.$$\n\n## Remarks\n\nThe step $p_k = x_{k+1} - x_k$ is the [[least squares|0BYTQ4X1HC]] solution of the linearized problem $\\min_p \\norm{r(x_k) + J(x_k) p}_2$.\nIn practice one computes it by a QR factorization of $J(x_k)$ rather than by forming $J^T J$.\nThe Hessian of $S$ is $J^T J + \\sum_i r_i \\nabla^2 r_i$.\nGauss–Newton drops the second term, so near a zero-residual solution it behaves like [[Newton's method|07EXVNF9VP]], which gives the quadratic rate.\n\nIf $r(x^*) \\ne 0$, the dropped term need not vanish, and the convergence is then in general only linear.\nFor large residuals or strongly nonlinear $r_i$, the iteration may diverge even from good starting points.\nA line search or the [[Levenberg–Marquardt algorithm|TH54SPVN1N]] makes the method robust.\nExample: fitting $y = e^{c t}$ to the data $(0, 1)$, $(1, e)$ has zero residual at $c = 1$, so Gauss–Newton converges quadratically from any $c_0$ near $1$.\n\n## Explanations\n\n### L1 human\n\nMany fitting problems ask for parameters that make a nonlinear model match data as closely as possible in the least squares sense.\nGauss–Newton replaces the model by its linear approximation at the current guess, solves the resulting linear least squares problem, and repeats.\nWhen the model can fit the data exactly, it converges very fast near the answer.\n\n### L2 human\n\nThe method needs only first derivatives of the residuals, unlike Newton's method for $S$, which needs second derivatives.\nIt is the basic method of [[nonlinear regression|C7E69V030F]] and of many calibration and inverse problems.\nThe rank condition on $J(x^*)$ is needed: without it the linearized problems have no unique solution, and the convergence theory fails.\n",
  "commit": "ff8a40749dc16545e3d2fc637f6b0196e81d0bfa",
  "front": {
    "about": [
      "C7E69V030F"
    ],
    "kind": "statement",
    "msc": [
      "65K10",
      "90C30",
      "62J02"
    ],
    "name": "Gauss–Newton algorithm",
    "role": "theorem",
    "same_as": {
      "wikidata": "Q1496373",
      "wikipedia": "Gauss–Newton algorithm"
    },
    "sources": [
      {
        "url": "https://en.wikipedia.org/wiki/Gauss%E2%80%93Newton_algorithm"
      }
    ],
    "tag": "CYV4T43FQJ",
    "uses": [
      "047NDTEPCV",
      "0BYTQ4X1HC",
      "07EXVNF9VP",
      "0QNVQ4VD9T",
      "YWZQC5E54B"
    ]
  },
  "kind": "statement",
  "link": "https://tamnd.github.io/math2/node/CYV4T43FQJ/",
  "links": [
    {
      "field": "about",
      "target": "C7E69V030F"
    },
    {
      "field": "uses",
      "target": "047NDTEPCV"
    },
    {
      "field": "uses",
      "target": "0BYTQ4X1HC"
    },
    {
      "field": "uses",
      "target": "07EXVNF9VP"
    },
    {
      "field": "uses",
      "target": "0QNVQ4VD9T"
    },
    {
      "field": "uses",
      "target": "YWZQC5E54B"
    }
  ],
  "markdown": "https://tamnd.github.io/math2/node/CYV4T43FQJ.md",
  "msc": [
    "65K10",
    "90C30",
    "62J02"
  ],
  "name": "Gauss–Newton algorithm",
  "prov": null,
  "role": "theorem",
  "source": "https://github.com/tamnd/math2/blob/main/nodes/CYV/CYV4T43FQJ.md",
  "status": null,
  "tag": "CYV4T43FQJ",
  "uri": "m2://CYV4T43FQJ",
  "url": "https://tamnd.github.io/math2/wiki/gauss-newton-algorithm/",
  "writing": {
    "tag": "CYV4T43FQJ",
    "kind": "statement",
    "level": "W0",
    "next": [
      {
        "what": "an inventory row",
        "note": "the row of the inventory that the node accounts for",
        "have": 0,
        "want": 1
      },
      {
        "what": "a source with a locator",
        "have": 0,
        "want": 1
      }
    ]
  }
}