{
  "aliases": null,
  "backlinks": null,
  "body": "\n## Definition\n\nLet $(\\Omega, \\mathcal{F}, (\\mathcal{F}_t)_{t \\ge 0}, P)$ be a probability space with a [[filtration|HNK1Y62G7V]] that carries a $d$-dimensional [[Wiener process|22735FGJV6]] $W$.\nLet $T \u003e 0$ and let $U \\subseteq \\R^m$ be a nonempty closed set, the control set.\nLet $b \\colon \\R^n \\times U \\to \\R^n$ and $\\sigma \\colon \\R^n \\times U \\to \\R^{n \\times d}$ be continuous.\nAssume that they are [[Lipschitz|0QNVQ4VD9T]] in $x$ uniformly in $u$, and that $\\abs{b(x, u)} + \\abs{\\sigma(x, u)} \\le K (1 + \\abs{x} + \\abs{u})$ for a constant $K$.\nLet $f \\colon \\R^n \\times U \\to \\R$ and $g \\colon \\R^n \\to \\R$ be continuous with $\\abs{f(x, u)} \\le K (1 + \\abs{x}^2 + \\abs{u}^2)$ and $\\abs{g(x)} \\le K (1 + \\abs{x}^2)$.\n\n1. An *admissible control* is a progressively measurable process $u = (u_t)_{0 \\le t \\le T}$ with values in $U$ and $E \\int_0^T \\abs{u_t}^2 \\, dt \u003c \\infty$.\n2. For an admissible control $u$ and $x \\in \\R^n$, the *state* is the solution $X^u$ of the [[stochastic differential equation|B219G9SHJA]] $dX_t = b(X_t, u_t) \\, dt + \\sigma(X_t, u_t) \\, dW_t$ with $X_0 = x$.\n3. The *cost* of $u$ is $J(x; u) = E\\left[ \\int_0^T f(X^u_t, u_t) \\, dt + g(X^u_T) \\right]$, an [[expected value|0WQNRCNY3M]].\n\nThe *stochastic optimal control problem* is to find an admissible control $u^*$ with $J(x; u^*) = \\inf_u J(x; u)$, the infimum over all admissible controls.\nSuch a $u^*$ is an *optimal control*, and $V(x) = \\inf_u J(x; u)$ is the *value*.\n\n## Remarks\n\nUnder these hypotheses the state equation has a unique strong solution and the cost is finite.\nStarting the problem at time $t$ in state $x$ gives the value function $V(t, x)$.\nWhen $V$ is smooth enough, it solves the [[Hamilton–Jacobi–Bellman equation|SJP9HW82VX]]\n$$V_t + \\min_{u \\in U} \\left\\{ b(x, u) \\cdot \\nabla_x V + \\tfrac{1}{2} \\tr\\left( \\sigma \\sigma^T(x, u) D_x^2 V \\right) + f(x, u) \\right\\} = 0, \\qquad V(T, x) = g(x).$$\nWith $\\sigma = 0$ the problem is a deterministic optimal control problem.\n\nExample: $n = m = d = 1$, $U = \\R$, $b(x, u) = u$, $\\sigma = 1$, $f(x, u) = u^2 / 2$ and $g(x) = x^2 / 2$.\nThe function $V(t, x) = \\dfrac{x^2}{2 (1 + T - t)} + \\dfrac{1}{2} \\ln(1 + T - t)$ satisfies $V(T, x) = x^2 / 2$.\nThe minimum over $u$ of $u V_x + u^2 / 2$ is $-\\frac{1}{2} V_x^2$, attained at $u = -V_x$.\nWith $s = 1 + T - t$, one has $V_t = \\frac{x^2}{2 s^2} - \\frac{1}{2 s}$, $\\frac{1}{2} V_x^2 = \\frac{x^2}{2 s^2}$ and $\\frac{1}{2} V_{xx} = \\frac{1}{2 s}$, so the equation holds.\nThe optimal control is the feedback $u_t = -X_t / (1 + T - t)$.\n\nNon-example: the process $u_t = W_T$ for all $t$ is not admissible, because it is not adapted: it uses the future of the noise.\n\n## Explanations\n\n### L1 human\n\nThe controller picks its action at each time using only what it has seen so far.\nThe noise makes the cost random, so one minimises its average.\n",
  "commit": "ff8a40749dc16545e3d2fc637f6b0196e81d0bfa",
  "front": {
    "defines": "B24T6DQCDR",
    "kind": "definition",
    "msc": [
      "93E20",
      "49L20"
    ],
    "name": "Stochastic optimal control",
    "sources": [
      {
        "url": "https://en.wikipedia.org/wiki/Stochastic_control"
      }
    ],
    "tag": "6KAVKR1HC8",
    "uses": [
      "B219G9SHJA",
      "22735FGJV6",
      "HNK1Y62G7V",
      "0WQNRCNY3M",
      "0QNVQ4VD9T",
      "SJP9HW82VX"
    ]
  },
  "kind": "definition",
  "link": "https://tamnd.github.io/math2/node/6KAVKR1HC8/",
  "links": [
    {
      "field": "defines",
      "target": "B24T6DQCDR"
    },
    {
      "field": "uses",
      "target": "B219G9SHJA"
    },
    {
      "field": "uses",
      "target": "22735FGJV6"
    },
    {
      "field": "uses",
      "target": "HNK1Y62G7V"
    },
    {
      "field": "uses",
      "target": "0WQNRCNY3M"
    },
    {
      "field": "uses",
      "target": "0QNVQ4VD9T"
    },
    {
      "field": "uses",
      "target": "SJP9HW82VX"
    }
  ],
  "markdown": "https://tamnd.github.io/math2/node/6KAVKR1HC8.md",
  "msc": [
    "93E20",
    "49L20"
  ],
  "name": "Stochastic optimal control",
  "prov": null,
  "role": "",
  "source": "https://github.com/tamnd/math2/blob/main/nodes/6KA/6KAVKR1HC8.md",
  "status": null,
  "tag": "6KAVKR1HC8",
  "uri": "m2://6KAVKR1HC8",
  "url": "https://tamnd.github.io/math2/wiki/stochastic-optimal-control-definition/",
  "writing": {
    "tag": "6KAVKR1HC8",
    "kind": "definition",
    "level": "W0",
    "next": [
      {
        "what": "an inventory row",
        "note": "the row of the inventory that the node accounts for",
        "have": 0,
        "want": 1
      },
      {
        "what": "a source with a locator",
        "have": 0,
        "want": 1
      }
    ]
  }
}