Chapter 5. Forward Mode Automatic Differentiation

Forward mode automatic differentiation computes derivatives by carrying two values through a program at the same time: the ordinary value and its tangent. The ordinary value...

9 items

Section	Title
1	Chapter 5. Forward Mode Automatic Differentiation
2	Dual Numbers
3	Forward Evaluation Rules
4	Jacobian-Vector Products
5	Complexity Analysis
6	Higher-Dimensional Tangent Spaces
7	Efficient Seeding Strategies
8	Sparse Forward Methods
9	Case Studies

Writes › Book › Auto Diff › Chapter 5. Forward Mode Automatic Differentiation ›

Chapter 5. Forward Mode Automatic Differentiation

Forward mode automatic differentiation computes derivatives by carrying two values through a program at the same time: the ordinary value and its tangent. The ordinary value...

Writes › Book › Auto Diff › Chapter 5. Forward Mode Automatic Differentiation ›

Dual Numbers

Dual numbers give forward mode automatic differentiation a compact algebraic form. Instead of storing a value and a tangent as two unrelated fields, we package them into one...

Writes › Book › Auto Diff › Chapter 5. Forward Mode Automatic Differentiation ›

Forward Evaluation Rules

Forward mode automatic differentiation works by replacing each primitive operation with an extended operation on pairs:

Writes › Book › Auto Diff › Chapter 5. Forward Mode Automatic Differentiation ›

Jacobian-Vector Products

The natural output of forward mode automatic differentiation is a Jacobian-vector product. Instead of constructing the full Jacobian matrix explicitly, forward mode computes...

Writes › Book › Auto Diff › Chapter 5. Forward Mode Automatic Differentiation ›

Complexity Analysis

Forward mode automatic differentiation has a simple cost model. It evaluates the original program and, at the same time, evaluates the tangent program. Each primitive...

Writes › Book › Auto Diff › Chapter 5. Forward Mode Automatic Differentiation ›

Higher-Dimensional Tangent Spaces

So far, forward mode has propagated a single tangent direction:

Writes › Book › Auto Diff › Chapter 5. Forward Mode Automatic Differentiation ›

Efficient Seeding Strategies

Forward mode automatic differentiation computes Jacobian-vector products:

Writes › Book › Auto Diff › Chapter 5. Forward Mode Automatic Differentiation ›

Sparse Forward Methods

Many real-world Jacobians are sparse. Most derivative entries are zero because outputs depend only on small subsets of inputs.

Writes › Book › Auto Diff › Chapter 5. Forward Mode Automatic Differentiation ›

Case Studies

Forward mode automatic differentiation appears in many numerical systems where directional derivatives, local sensitivities, or small parameter sets are important. This...