{
  "aliases": null,
  "backlinks": null,
  "body": "\n## Statement\n\nLet $D \\subseteq \\R^n$ be open, let $r \\colon D \\to \\R^m$ be continuously differentiable with [[Jacobian matrix|047NDTEPCV]] $J(x)$, and let $S(x) = \\tfrac12 \\norm{r(x)}_2^2$.\nFor $x \\in D$ and a damping parameter $\\lambda \u003e 0$, the *Levenberg–Marquardt step* $\\delta = \\delta(\\lambda)$ is the solution of\n$$\\bigl( J(x)^T J(x) + \\lambda I \\bigr) \\delta = -J(x)^T r(x).$$\nWrite $J = J(x)$, $r = r(x)$ and $g = \\nabla S(x) = J^T r$.\nThen:\n\n1. $J^T J + \\lambda I$ is [[positive definite|0KX66CZMBB]], so $\\delta(\\lambda)$ exists and is unique;\n2. if $g \\ne 0$, then $g^T \\delta(\\lambda) \u003c 0$, so $\\delta(\\lambda)$ is a descent direction for $S$;\n3. $\\delta(\\lambda)$ minimizes $\\norm{r + J p}_2$ over all $p$ with $\\norm{p}_2 \\le \\norm{\\delta(\\lambda)}_2$;\n4. if $J$ has full column rank, then $\\delta(\\lambda)$ tends to the [[Gauss–Newton|CYV4T43FQJ]] step as $\\lambda \\to 0$;\n5. $\\lambda \\, \\delta(\\lambda) \\to -g$ as $\\lambda \\to \\infty$.\n\n## Remarks\n\nPart 1 holds because $p^T (J^T J + \\lambda I) p = \\norm{J p}^2 + \\lambda \\norm{p}^2 \u003e 0$ for $p \\ne 0$.\nPart 2 follows: $g^T \\delta = -g^T (J^T J + \\lambda I)^{-1} g \u003c 0$.\nFor part 3, $\\delta$ satisfies the optimality conditions of the ball-constrained problem with multiplier $\\lambda \\ge 0$, and the problem is convex.\nSo the method is a trust region method for the Gauss–Newton model, with $\\lambda$ controlling the radius.\nParts 4 and 5 show that it interpolates between Gauss–Newton steps (small $\\lambda$) and short steepest descent steps (large $\\lambda$).\n\nThe algorithm updates $\\lambda$ at each step.\nIf the step reduces $S$ enough, it is accepted and $\\lambda$ is decreased; otherwise it is rejected and $\\lambda$ is increased.\nMarquardt proposed replacing $\\lambda I$ by $\\lambda \\, \\mathrm{diag}(J^T J)$, which makes the step invariant under rescaling of the variables.\nExample: with $m = n = 1$, $r(x) = x - 1$ and $x = 3$, one has $J = 1$, $g = 2$ and $\\delta(\\lambda) = -2/(1 + \\lambda)$, which is the full Gauss–Newton step $-2$ at $\\lambda = 0$ and shrinks as $\\lambda$ grows.\n\n## Explanations\n\n### L1 human\n\nGauss–Newton is fast near a solution but can jump too far when the guess is poor.\nLevenberg–Marquardt adds a damping term that shortens the step and turns it toward the downhill direction.\nWhen things go well the damping is reduced and the method becomes Gauss–Newton; when they go badly it is increased.\n\n### L2 human\n\nThe method is due to Kenneth Levenberg (1944) and Donald Marquardt (1963).\nIt is the standard algorithm for small and medium nonlinear least squares problems, such as curve fitting and camera calibration.\nIts damped normal equations are the same equations as in Tikhonov regularization of the linearized problem.\n",
  "commit": "ff8a40749dc16545e3d2fc637f6b0196e81d0bfa",
  "front": {
    "about": [
      "C7E69V030F"
    ],
    "kind": "statement",
    "msc": [
      "65K10",
      "90C30",
      "62J02"
    ],
    "name": "Levenberg–Marquardt algorithm",
    "role": "proposition",
    "same_as": {
      "wikidata": "Q1426494",
      "wikipedia": "Levenberg–Marquardt algorithm"
    },
    "sources": [
      {
        "url": "https://en.wikipedia.org/wiki/Levenberg%E2%80%93Marquardt_algorithm"
      }
    ],
    "tag": "TH54SPVN1N",
    "uses": [
      "047NDTEPCV",
      "0BYTQ4X1HC",
      "0KX66CZMBB",
      "CYV4T43FQJ"
    ]
  },
  "kind": "statement",
  "link": "https://tamnd.github.io/math2/node/TH54SPVN1N/",
  "links": [
    {
      "field": "about",
      "target": "C7E69V030F"
    },
    {
      "field": "uses",
      "target": "047NDTEPCV"
    },
    {
      "field": "uses",
      "target": "0BYTQ4X1HC"
    },
    {
      "field": "uses",
      "target": "0KX66CZMBB"
    },
    {
      "field": "uses",
      "target": "CYV4T43FQJ"
    }
  ],
  "markdown": "https://tamnd.github.io/math2/node/TH54SPVN1N.md",
  "msc": [
    "65K10",
    "90C30",
    "62J02"
  ],
  "name": "Levenberg–Marquardt algorithm",
  "prov": null,
  "role": "proposition",
  "source": "https://github.com/tamnd/math2/blob/main/nodes/TH5/TH54SPVN1N.md",
  "status": null,
  "tag": "TH54SPVN1N",
  "uri": "m2://TH54SPVN1N",
  "url": "https://tamnd.github.io/math2/wiki/levenberg-marquardt-algorithm/",
  "writing": {
    "tag": "TH54SPVN1N",
    "kind": "statement",
    "level": "W0",
    "next": [
      {
        "what": "an inventory row",
        "note": "the row of the inventory that the node accounts for",
        "have": 0,
        "want": 1
      },
      {
        "what": "a source with a locator",
        "have": 0,
        "want": 1
      }
    ]
  }
}