{
  "aliases": null,
  "backlinks": null,
  "body": "\n## Definition\n\nLet $U \\subseteq \\R^n$ be open and let $f \\colon U \\to \\R^m$ be given by an *evaluation trace*.\nIts first variables $v_{1-n}, \\ldots, v_0$ are the inputs $x_1, \\ldots, x_n$.\nThe others are given for $k = 1, \\ldots, \\ell$ by\n$$v_k = \\varphi_k\\big( (v_j)_{j \\in P_k} \\big),$$\nwhere $P_k \\subseteq \\set{1 - n, \\ldots, k - 1}$ and each $\\varphi_k$ is a differentiable elementary function, such as $+$, $\\times$, $\\sin$ or $\\exp$.\nThe outputs $y_1, \\ldots, y_m$ are some of the $v_k$.\n\n1. *Forward mode* takes a direction $\\dot x \\in \\R^n$, sets $\\dot v_{i - n} = \\dot x_i$, and computes along with each $v_k$ the value $\\dot v_k = \\sum_{j \\in P_k} \\frac{\\partial \\varphi_k}{\\partial v_j} \\dot v_j$.\n2. *Reverse mode* first evaluates the trace and takes $\\bar y \\in \\R^m$.\nIt sets the adjoint $\\bar v$ of each output to its entry of $\\bar y$ and all other adjoints to $0$.\nThen for $k = \\ell, \\ell - 1, \\ldots, 1$ it adds $\\bar v_k \\frac{\\partial \\varphi_k}{\\partial v_j}$ to $\\bar v_j$ for each $j \\in P_k$.\n\n*Automatic differentiation* is the computation of derivatives of $f$ by these propagation rules.\nForward mode returns $\\dot y = J_f(x) \\dot x$, where $J_f(x)$ is the [[Jacobian matrix|047NDTEPCV]].\nReverse mode returns $\\bar x = J_f(x)^{T} \\bar y$, where $\\bar x_i = \\bar v_{i-n}$.\n\n## Remarks\n\nExample: $f(x_1, x_2) = x_1 x_2 + \\sin x_1$ has the trace $v_1 = v_{-1} v_0$, $v_2 = \\sin v_{-1}$, $v_3 = v_1 + v_2$.\nForward mode at $(x_1, x_2) = (1, 2)$ with $\\dot x = (1, 0)$ gives $\\dot v_1 = 2$, $\\dot v_2 = \\cos 1$, $\\dot v_3 = 2 + \\cos 1$, which is the [[partial derivative|09EGC6F8PT]] $\\partial f / \\partial x_1 = x_2 + \\cos x_1$ at $(1, 2)$.\nReverse mode with $\\bar y = 1$ gives $\\bar v_1 = \\bar v_2 = 1$, then $\\bar v_0 = v_{-1} = 1$ and $\\bar v_{-1} = v_0 + \\cos v_{-1} = 2 + \\cos 1$, the full gradient in one backward sweep.\nNon-example: the difference quotient $\\frac{f(x + h e_1) - f(x)}{h}$ is not automatic differentiation; it does not use the trace and has a truncation error of order $h$.\nForward mode can be described with dual numbers $a + b\\varepsilon$, $\\varepsilon^2 = 0$: evaluating the program on $x + \\dot x \\varepsilon$ gives $f(x) + (J_f(x) \\dot x) \\varepsilon$.\n\n## Explanations\n\n### L1 human\n\nAutomatic differentiation follows the program step by step and differentiates each step by the rules of calculus.\nForward mode answers how all outputs change when one input direction changes; reverse mode answers how one output depends on all inputs at once.\n",
  "commit": "ff8a40749dc16545e3d2fc637f6b0196e81d0bfa",
  "front": {
    "defines": "BW7FEZ8WM1",
    "kind": "definition",
    "msc": [
      "65D25",
      "68W30"
    ],
    "name": "Automatic differentiation",
    "sources": [
      {
        "url": "https://en.wikipedia.org/wiki/Automatic_differentiation"
      }
    ],
    "tag": "KDZN92Z8JR",
    "uses": [
      "0V1EJCT8G6",
      "09EGC6F8PT",
      "047NDTEPCV"
    ]
  },
  "kind": "definition",
  "link": "https://tamnd.github.io/math2/node/KDZN92Z8JR/",
  "links": [
    {
      "field": "defines",
      "target": "BW7FEZ8WM1"
    },
    {
      "field": "uses",
      "target": "0V1EJCT8G6"
    },
    {
      "field": "uses",
      "target": "09EGC6F8PT"
    },
    {
      "field": "uses",
      "target": "047NDTEPCV"
    }
  ],
  "markdown": "https://tamnd.github.io/math2/node/KDZN92Z8JR.md",
  "msc": [
    "65D25",
    "68W30"
  ],
  "name": "Automatic differentiation",
  "prov": null,
  "role": "",
  "source": "https://github.com/tamnd/math2/blob/main/nodes/KDZ/KDZN92Z8JR.md",
  "status": null,
  "tag": "KDZN92Z8JR",
  "uri": "m2://KDZN92Z8JR",
  "url": "https://tamnd.github.io/math2/wiki/automatic-differentiation-definition/",
  "writing": {
    "tag": "KDZN92Z8JR",
    "kind": "definition",
    "level": "W0",
    "next": [
      {
        "what": "an inventory row",
        "note": "the row of the inventory that the node accounts for",
        "have": 0,
        "want": 1
      },
      {
        "what": "a source with a locator",
        "have": 0,
        "want": 1
      }
    ]
  }
}