Dynamic programming in continuous time

Definition

Consider a finite-horizon optimal control problem on $[0, T]$ with dynamics $y' = f(s, y, u)$, controls with values in a set $U$, running cost $L$ and terminal cost $g$. Let $V$ be its value function. The dynamic programming principle for the problem is the identity, for all $0 \le t \le s \le T$ and $x \in \R^n$, $$V(t, x) = \inf_{u} \Big( \int_t^s L(r, y(r), u(r)) \, dr + V(s, y(s)) \Big).$$ Here the infimum is over measurable controls $u \colon [t, s] \to U$, and $y$ is the trajectory with $y(t) = x$. Dynamic programming in continuous time is the method that derives from this principle a partial differential equation for $V$. That equation is the Hamilton–Jacobi–Bellman equation $V_t + \min_{u \in U} \big( L(t, x, u) + \nabla_x V \cdot f(t, x, u) \big) = 0$ with $V(T, x) = g(x)$. One then obtains optimal controls by choosing, at each $(t, x)$, a minimizer $u$ of this expression.