Calculus of variations and optimal control; optimization Dynamic programming and Hamilton-Jacobi theory Theorem

Verification theorem

Builds on Value function Dynamic programming in continuous time Hamilton–Jacobi–Bellman equation Control system Linear-quadratic regulator Viscosity solution of the Hamilton–Jacobi–Bellman equation

Statement

Let $U \subseteq \R^m$ be a nonempty compact set and $T > 0$ . Let $f \colon [0, T] \times \R^n \times U \to \R^n$ be continuous and Lipschitz in $x$ uniformly in $(t, u)$ , and let $L$ and $g$ be continuous. Let $V$ be the value function of the problem of minimizing $\int_t^T L(s, y, u) \, ds + g(y(T))$ subject to $y' = f(s, y, u)$ , $y(t) = x$ . Let $W \colon [0, T] \times \R^n \to \R$ be continuous, of class $C^1$ on $(0, T) \times \R^n$ , with $W(T, x) = g(x)$ . Assume that $W$ solves the Hamilton–Jacobi–Bellman equation $W_t(t, x) + \min_{u \in U} \big( L(t, x, u) + \nabla_x W(t, x) \cdot f(t, x, u) \big) = 0 \quad \text{on } (0, T) \times \R^n.$ Then:

$W(t, x) \le V(t, x)$ for all $(t, x) \in [0, T] \times \R^n$ .
Let $u^*$ be a measurable control on $[t, T]$ with trajectory $y^*$ from $(t, x)$ . Suppose that for almost every $s \in [t, T]$ the value $u^*(s)$ attains the minimum in the equation at $(s, y^*(s))$ . Then $u^*$ is optimal from $(t, x)$ , and $W(t, x) = V(t, x)$ .

Remarks

The proof is a chain rule computation. For any control $u$ with trajectory $y$ , the function $s \mapsto W(s, y(s))$ is absolutely continuous with derivative $W_t + \nabla_x W \cdot f(s, y, u) \ge -L(s, y, u)$ by the equation. Integrating from $t$ to $T$ gives $g(y(T)) - W(t, x) \ge -\int_t^T L \, ds$ , that is, $W(t, x) \le J(t, x; u)$ . For $u^*$ the inequality is an equality at almost every time, so $J(t, x; u^*) = W(t, x) \le V(t, x)$ . Example: for the linear-quadratic regulator, $W(t, x) = x^{T} P(t) x$ with $P$ the solution of the Riccati differential equation solves the equation with $U = \R^m$ . The minimum is attained at $u = -R^{-1} B^{T} P(t) x$ , so this feedback is optimal; the proof adapts to the unbounded control set. The smoothness of $W$ is essential: value functions are often not differentiable, and then one needs the theory of viscosity solutions.

Going deeper

For a student

The theorem is the sufficiency half of dynamic programming. It is the standard way to prove that a feedback law found by solving the HJB equation, for example the linear feedback of the linear-quadratic regulator, is truly optimal. Pontryagin's maximum principle gives only necessary conditions, so the verification theorem complements it. When the value function is not smooth, versions of the theorem use viscosity solutions or generalized gradients.

Reading path

To understand this page from the ground up, read these 14 pages in order. The foundations come first.

Show the path as a graph

Each page uses the pages above it. The colour of the bar on a box shows how strong the evidence is, and the box also gives it as text. The list has the same pages.

no evidence level

Algebraic Riccati equation concept
Compact set concept
Control system concept
Dynamic programming in continuous time concept
Hamilton–Jacobi–Bellman equation concept
Optimal control concept
Positive definite matrix concept
Positive semidefinite matrix concept
Riccati differential equation of optimal control concept
Value function concept
Viscosity solution concept
Linear-quadratic regulator problem
Viscosity solution of the Hamilton–Jacobi–Bellman equation statement
Verification theorem statement

Tag	`T5BEKMG5W8`
URI	`m2://T5BEKMG5W8`
Kind	statement, theorem
Writing	W0 stub
MSC	49L20, 49L12, 49K15
Status	This copy of the site has no build data. Run `m2 status T5BEKMG5W8`.
About	Value function, Hamilton–Jacobi–Bellman equation
Uses	Value function, Dynamic programming in continuous time, Hamilton–Jacobi–Bellman equation, Control system, Linear-quadratic regulator, Viscosity solution of the Hamilton–Jacobi–Bellman equation