{
  "aliases": null,
  "backlinks": null,
  "body": "\n## Definition\n\nLet $F \\colon \\R^n \\times \\R^m \\to \\R^n$, $\\ell \\colon \\R^n \\times \\R^m \\to \\R$ and $\\ell_f \\colon \\R^n \\to \\R$ be twice continuously differentiable, and let $N \\ge 1$.\nConsider the problem of minimising $\\sum_{k=0}^{N-1} \\ell(x_k, u_k) + \\ell_f(x_N)$ over $u_0, \\dots, u_{N-1} \\in \\R^m$, where $x_{k+1} = F(x_k, u_k)$ and $x_0$ is given.\nSubscripts denote partial derivatives, $\\nabla^2$ a [[Hessian|0693Y2METT]] and $\\nabla$ a [[gradient|0ABK9V9Y64]].\nOne iteration of *differential dynamic programming* starts from controls $\\bar u_k$ with states $\\bar x_k$ and does the following.\n\n1. Backward pass: set $V_x = \\nabla \\ell_f(\\bar x_N)$ and $V_{xx} = \\nabla^2 \\ell_f(\\bar x_N)$. For $k = N - 1, \\dots, 0$, take all derivatives at $(\\bar x_k, \\bar u_k)$ and $V_x, V_{xx}$ from step $k + 1$. Compute\n$$Q_x = \\ell_x + F_x^T V_x, \\quad Q_u = \\ell_u + F_u^T V_x,$$\n$$Q_{xx} = \\ell_{xx} + F_x^T V_{xx} F_x + V_x \\cdot F_{xx}, \\quad Q_{uu} = \\ell_{uu} + F_u^T V_{xx} F_u + V_x \\cdot F_{uu}, \\quad Q_{ux} = \\ell_{ux} + F_u^T V_{xx} F_x + V_x \\cdot F_{ux},$$\nwhere $V_x \\cdot F_{xx}$ is the matrix $\\sum_i (V_x)_i \\nabla_x^2 F_i$ and similarly for the others. If $Q_{uu}$ is positive definite, set $d_k = -Q_{uu}^{-1} Q_u$, $K_k = -Q_{uu}^{-1} Q_{ux}$, and update $V_x \\leftarrow Q_x - Q_{ux}^T Q_{uu}^{-1} Q_u$ and $V_{xx} \\leftarrow Q_{xx} - Q_{ux}^T Q_{uu}^{-1} Q_{ux}$.\n2. Forward pass: set $x_0^{\\text{new}} = x_0$ and, for $k = 0, \\dots, N - 1$, set $u_k^{\\text{new}} = \\bar u_k + d_k + K_k (x_k^{\\text{new}} - \\bar x_k)$ and $x_{k+1}^{\\text{new}} = F(x_k^{\\text{new}}, u_k^{\\text{new}})$.\n\nThe new controls replace $\\bar u_k$, and the iteration is repeated until the corrections $d_k$ are small.\n\n## Remarks\n\nThe function $Q$ is the second-order expansion of $\\delta x, \\delta u \\mapsto \\ell(\\bar x_k + \\delta x, \\bar u_k + \\delta u) + V_{k+1}(F(\\bar x_k + \\delta x, \\bar u_k + \\delta u))$, where $V_{k+1}$ is the quadratic model of the cost-to-go.\nThe correction $\\delta u = d_k + K_k \\delta x$ minimises this quadratic model in $\\delta u$, and the update of $V_x, V_{xx}$ gives the model of the minimum.\nDropping the terms with second derivatives of $F$ gives the iterative linear-quadratic regulator.\nIn practice one scales $d_k$ by a step size and adds a multiple of the identity to $Q_{uu}$ for robustness.\n\nExample: if $F(x, u) = A x + B u$ and $\\ell$, $\\ell_f$ are quadratic forms with $\\ell_{uu}$ positive definite, the expansions are exact.\nThen the backward pass is the discrete Riccati recursion of the linear-quadratic regulator, and one iteration gives the optimal controls from any starting guess.\n\nNon-example: take $\\ell = -u^2$, $F(x, u) = x$ and $\\ell_f = 0$, so that $Q_{uu} = -2$ is not positive definite.\nThe quadratic model has no minimiser in $\\delta u$, and the step of clause 1 is not defined.\n\n## Explanations\n\n### L1 human\n\nEach iteration fits quadratic models of the costs around the current trajectory, solves them backward in time, and then simulates the improved controls forward.\n",
  "commit": "ff8a40749dc16545e3d2fc637f6b0196e81d0bfa",
  "front": {
    "defines": "5D0J26E619",
    "kind": "definition",
    "msc": [
      "49M15",
      "49L20",
      "93C55"
    ],
    "name": "Differential dynamic programming",
    "sources": [
      {
        "url": "https://en.wikipedia.org/wiki/Differential_dynamic_programming"
      }
    ],
    "tag": "WA3974X06N",
    "uses": [
      "0693Y2METT",
      "0ABK9V9Y64",
      "07EXVNF9VP"
    ]
  },
  "kind": "definition",
  "link": "https://tamnd.github.io/math2/node/WA3974X06N/",
  "links": [
    {
      "field": "defines",
      "target": "5D0J26E619"
    },
    {
      "field": "uses",
      "target": "0693Y2METT"
    },
    {
      "field": "uses",
      "target": "0ABK9V9Y64"
    },
    {
      "field": "uses",
      "target": "07EXVNF9VP"
    }
  ],
  "markdown": "https://tamnd.github.io/math2/node/WA3974X06N.md",
  "msc": [
    "49M15",
    "49L20",
    "93C55"
  ],
  "name": "Differential dynamic programming",
  "prov": null,
  "role": "",
  "source": "https://github.com/tamnd/math2/blob/main/nodes/WA3/WA3974X06N.md",
  "status": null,
  "tag": "WA3974X06N",
  "uri": "m2://WA3974X06N",
  "url": "https://tamnd.github.io/math2/wiki/differential-dynamic-programming-definition/",
  "writing": {
    "tag": "WA3974X06N",
    "kind": "definition",
    "level": "W0",
    "next": [
      {
        "what": "an inventory row",
        "note": "the row of the inventory that the node accounts for",
        "have": 0,
        "want": 1
      },
      {
        "what": "a source with a locator",
        "have": 0,
        "want": 1
      }
    ]
  }
}