Chapter 4. Core Theory of Automatic Differentiation

Automatic differentiation is built on a simple observation: a complicated derivative can be computed by composing many small local derivatives. Instead of manipulating a full...

10 items

Section	Title
1	Chapter 4. Core Theory of Automatic Differentiation
2	Elementary Operations
3	Forward Accumulation
4	Reverse Accumulation
5	Mixed-Mode Differentiation
6	Computational Complexity
7	Numerical Exactness up to Floating Point
8	AD as Program Transformation
9	Summary and Synthesis
10	Tangent Propagation

Writes › Book › Auto Diff › Chapter 4. Core Theory of Automatic Differentiation ›

Chapter 4. Core Theory of Automatic Differentiation

Automatic differentiation is built on a simple observation: a complicated derivative can be computed by composing many small local derivatives. Instead of manipulating a full...

Writes › Book › Auto Diff › Chapter 4. Core Theory of Automatic Differentiation ›

Elementary Operations

Automatic differentiation reduces differentiation to a finite collection of elementary operations. Every program, regardless of complexity, is decomposed into primitive...

Writes › Book › Auto Diff › Chapter 4. Core Theory of Automatic Differentiation ›

Forward Accumulation

Forward accumulation is the forward-mode form of automatic differentiation. It propagates derivative information in the same order as ordinary program evaluation. Each...

Writes › Book › Auto Diff › Chapter 4. Core Theory of Automatic Differentiation ›

Reverse Accumulation

Reverse accumulation is the reverse-mode form of automatic differentiation. It propagates derivative information backward from outputs to inputs.

Writes › Book › Auto Diff › Chapter 4. Core Theory of Automatic Differentiation ›

Mixed-Mode Differentiation

Mixed-mode differentiation combines forward accumulation and reverse accumulation in the same derivative computation. It is used when neither pure forward mode nor pure...

Writes › Book › Auto Diff › Chapter 4. Core Theory of Automatic Differentiation ›

Computational Complexity

Automatic differentiation is fundamentally a computational technique. Its practical importance comes from the fact that derivatives can often be computed with asymptotic cost...

Writes › Book › Auto Diff › Chapter 4. Core Theory of Automatic Differentiation ›

Numerical Exactness up to Floating Point

Automatic differentiation computes derivatives exactly with respect to the executed floating point program. This distinguishes AD from numerical differentiation, which...

Writes › Book › Auto Diff › Chapter 4. Core Theory of Automatic Differentiation ›

AD as Program Transformation

Automatic differentiation can be understood as a transformation from one program into another program.

Writes › Book › Auto Diff › Chapter 4. Core Theory of Automatic Differentiation ›

Summary and Synthesis

Automatic differentiation is a method for computing derivatives by transforming programs into derivative-propagating computations. It does not approximate derivatives...

Writes › Book › Auto Diff › Chapter 4. Core Theory of Automatic Differentiation ›

Tangent Propagation

Forward mode automatic differentiation computes derivatives by propagating tangent values alongside ordinary values. The ordinary value is called the primal. The derivative...