Chapter 8. Higher-Order Differentiation

First derivatives describe local rate of change. Second derivatives describe how that rate of change itself changes. In optimization, this is curvature. In dynamics, it is...

9 items

Section	Title
1	Chapter 8. Higher-Order Differentiation
2	Hessian Computation
3	Hessian-Vector Products
4	Higher-Order Reverse Mode
5	Nested AD
6	Taylor Mode AD
7	Efficient Higher-Order Methods
8	Perturbation Confusion
9	Complexity of Higher Orders

Writes › Book › Auto Diff › Chapter 8. Higher-Order Differentiation ›

Chapter 8. Higher-Order Differentiation

First derivatives describe local rate of change. Second derivatives describe how that rate of change itself changes. In optimization, this is curvature. In dynamics, it is...

Writes › Book › Auto Diff › Chapter 8. Higher-Order Differentiation ›

Hessian Computation

For a scalar function

Writes › Book › Auto Diff › Chapter 8. Higher-Order Differentiation ›

Hessian-Vector Products

A Hessian-vector product computes

Writes › Book › Auto Diff › Chapter 8. Higher-Order Differentiation ›

Higher-Order Reverse Mode

Reverse mode is efficient for scalar-output functions because it propagates one adjoint backward through the computation and produces a full gradient. For

Writes › Book › Auto Diff › Chapter 8. Higher-Order Differentiation ›

Nested AD

Nested automatic differentiation means applying automatic differentiation inside another automatic differentiation computation.

Writes › Book › Auto Diff › Chapter 8. Higher-Order Differentiation ›

Taylor Mode AD

Taylor mode automatic differentiation computes derivatives by propagating truncated Taylor series through a program.

Writes › Book › Auto Diff › Chapter 8. Higher-Order Differentiation ›

Efficient Higher-Order Methods

Higher-order derivatives contain rich geometric information, but naïve computation quickly becomes impractical.

Writes › Book › Auto Diff › Chapter 8. Higher-Order Differentiation ›

Perturbation Confusion

Perturbation confusion is a correctness bug that appears in nested automatic differentiation, especially nested forward mode. It happens when two derivative computations...

Writes › Book › Auto Diff › Chapter 8. Higher-Order Differentiation ›

Complexity of Higher Orders

Higher-order automatic differentiation faces a fundamental problem: derivative structure grows combinatorially with order.