{
  "aliases": null,
  "backlinks": null,
  "body": "\n## Statement\n\nLet $A$ be a real $m \\times n$ matrix with nonzero rows $a_1^T, \\dots, a_m^T$, and let $b \\in \\R^m$ be such that the [[linear system|0GT4WWGWH2]] $A x = b$ has a solution.\nFor an index $i$, the *Kaczmarz step* maps $x \\in \\R^n$ to\n$$x + \\frac{b_i - \\ip{a_i}{x}}{\\norm{a_i}^2} \\, a_i,$$\nthe [[orthogonal projection|0B8KZRT28M]] of $x$ onto the hyperplane $\\set{y : \\ip{a_i}{y} = b_i}$.\nStart from any $x_0 \\in \\R^n$.\n\n1. (Cyclic version) Let $x_{k+1}$ be the Kaczmarz step from $x_k$ with $i = (k \\bmod m) + 1$. Then $x_k$ converges to a solution of $A x = b$. If $x_0 = 0$, the limit is the solution of least Euclidean norm.\n2. (Randomized version, Strohmer and Vershynin) Suppose $A$ has [[rank|0ZC9YTXZRY]] $n$, and let $x^*$ be the solution. Let $x_{k+1}$ be the Kaczmarz step from $x_k$ with $i$ drawn independently, equal to $j$ with probability $\\norm{a_j}^2 / \\norm{A}_F^2$. Then $\\mathbb{E} \\norm{x_k - x^*}^2 \\le \\left( 1 - \\kappa^{-2} \\right)^k \\norm{x_0 - x^*}^2$, where $\\kappa = \\norm{A}_F \\norm{A^{+}}_2$.\n\nHere $\\norm{\\cdot}$ is the Euclidean norm, $\\norm{A}_F$ is the Frobenius norm and $A^{+}$ is the pseudoinverse.\n\n## Remarks\n\nEach step is a projection onto a convex set that contains every solution.\nSo for every solution $z$, the distance $\\norm{x_k - z}$ never increases, which drives the convergence proof.\nEach step adds a multiple of a row $a_i$, so $x_k - x_0$ stays in the row space of $A$.\nThe solution of least norm is the only solution in the row space, which gives the claim for $x_0 = 0$.\nExample: for the system $x_1 = 1$, $x_2 = 1$ in $\\R^2$, two cyclic steps from $0$ give $(1, 0)$ and then the solution $(1, 1)$.\nFor orthogonal rows the method finishes after one sweep; for nearly parallel rows it zigzags and is slow.\nThe number $\\kappa$ is a scaled [[condition number|AB3S6MHJ8X]], and it measures this effect in the randomized bound.\n\n## Explanations\n\n### L1 human\n\nEach equation of a linear system describes a flat set, a line in the plane or a plane in space.\nThe Kaczmarz method moves the current point straight onto one of these sets, then onto the next, and so on.\nThe point gets closer to the common intersection at each move.\nThis simple idea is the algebraic reconstruction technique of computed tomography.\n\n### L2 human\n\nThe Kaczmarz method touches one row at a time, so it suits huge systems whose rows are generated on the fly.\nIt is a row-action [[iterative method|2RPE77ACER]], related to the Gauss–Seidel method applied to $A A^T y = b$ with $x = A^T y$.\nChoosing rows at random with probability proportional to their squared norms gives a clean linear rate in expectation.\nThis randomized version links the method to stochastic gradient descent for least squares.\n",
  "commit": "ff8a40749dc16545e3d2fc637f6b0196e81d0bfa",
  "front": {
    "about": [
      "2RPE77ACER"
    ],
    "kind": "statement",
    "msc": [
      "65F10",
      "65F20"
    ],
    "name": "Kaczmarz method",
    "role": "theorem",
    "same_as": {
      "wikidata": "Q288600",
      "wikipedia": "Kaczmarz method"
    },
    "sources": [
      {
        "url": "https://en.wikipedia.org/wiki/Kaczmarz_method"
      }
    ],
    "tag": "CSETD54YDT",
    "uses": [
      "0GT4WWGWH2",
      "0NB0AWGJBW",
      "0B8KZRT28M",
      "0ZC9YTXZRY",
      "AB3S6MHJ8X",
      "2RPE77ACER"
    ]
  },
  "kind": "statement",
  "link": "https://tamnd.github.io/math2/node/CSETD54YDT/",
  "links": [
    {
      "field": "about",
      "target": "2RPE77ACER"
    },
    {
      "field": "uses",
      "target": "0GT4WWGWH2"
    },
    {
      "field": "uses",
      "target": "0NB0AWGJBW"
    },
    {
      "field": "uses",
      "target": "0B8KZRT28M"
    },
    {
      "field": "uses",
      "target": "0ZC9YTXZRY"
    },
    {
      "field": "uses",
      "target": "AB3S6MHJ8X"
    },
    {
      "field": "uses",
      "target": "2RPE77ACER"
    }
  ],
  "markdown": "https://tamnd.github.io/math2/node/CSETD54YDT.md",
  "msc": [
    "65F10",
    "65F20"
  ],
  "name": "Kaczmarz method",
  "prov": null,
  "role": "theorem",
  "source": "https://github.com/tamnd/math2/blob/main/nodes/CSE/CSETD54YDT.md",
  "status": null,
  "tag": "CSETD54YDT",
  "uri": "m2://CSETD54YDT",
  "url": "https://tamnd.github.io/math2/wiki/kaczmarz-method/",
  "writing": {
    "tag": "CSETD54YDT",
    "kind": "statement",
    "level": "W0",
    "next": [
      {
        "what": "an inventory row",
        "note": "the row of the inventory that the node accounts for",
        "have": 0,
        "want": 1
      },
      {
        "what": "a source with a locator",
        "have": 0,
        "want": 1
      }
    ]
  }
}