Chapter 6. Reverse Mode Automatic Differentiation

Reverse mode automatic differentiation computes derivatives by propagating sensitivities backward through a computation. In forward mode, each intermediate value carries a...

10 items

Section Title
1 Chapter 6. Reverse Mode Automatic Differentiation
2 Reverse Computational Graphs
3 Vector-Jacobian Products
4 Reverse Accumulation Algorithms
5 Tape-Based Systems
6 Wengert Lists
7 Memory-Time Tradeoffs
8 Checkpointing
9 Reverse Mode in Deep Learning
10 Case Studies
Writes › Book › Auto Diff › Chapter 6. Reverse Mode Automatic Differentiation ›

Chapter 6. Reverse Mode Automatic Differentiation

Reverse mode automatic differentiation computes derivatives by propagating sensitivities backward through a computation. In forward mode, each intermediate value carries a...
autodiff book May 10, 2026
Writes › Book › Auto Diff › Chapter 6. Reverse Mode Automatic Differentiation ›

Reverse Computational Graphs

Reverse mode automatic differentiation operates on a computational graph. The forward pass evaluates the graph from inputs to outputs. The reverse pass traverses the same...
autodiff book May 10, 2026
Writes › Book › Auto Diff › Chapter 6. Reverse Mode Automatic Differentiation ›

Vector-Jacobian Products

Reverse mode automatic differentiation fundamentally computes vector-Jacobian products. The gradient of a scalar function is a special case of this more general operation.
autodiff book May 10, 2026
Writes › Book › Auto Diff › Chapter 6. Reverse Mode Automatic Differentiation ›

Reverse Accumulation Algorithms

Reverse accumulation is the operational core of reverse mode automatic differentiation. The forward pass evaluates a program and records dependency information. The reverse...
autodiff book May 10, 2026
Writes › Book › Auto Diff › Chapter 6. Reverse Mode Automatic Differentiation ›

Tape-Based Systems

Most reverse mode automatic differentiation systems require a mechanism for recording the forward computation so that the reverse pass can later traverse it backward. This...
autodiff book May 10, 2026
Writes › Book › Auto Diff › Chapter 6. Reverse Mode Automatic Differentiation ›

Wengert Lists

A Wengert list is a linear representation of a computation in which every intermediate result is assigned to a unique variable. It is one of the earliest and most influential...
autodiff book May 10, 2026
Writes › Book › Auto Diff › Chapter 6. Reverse Mode Automatic Differentiation ›

Memory-Time Tradeoffs

Reverse mode automatic differentiation is computationally efficient for scalar-output functions, but it has a major systems cost: it needs information from the forward pass...
autodiff book May 11, 2026
Writes › Book › Auto Diff › Chapter 6. Reverse Mode Automatic Differentiation ›

Checkpointing

Checkpointing is a technique for reducing the memory cost of reverse mode automatic differentiation by selectively storing intermediate states and recomputing missing values...
autodiff book May 11, 2026
Writes › Book › Auto Diff › Chapter 6. Reverse Mode Automatic Differentiation ›

Reverse Mode in Deep Learning

Reverse mode automatic differentiation is the mathematical and systems basis of backpropagation. In deep learning, the objective is usually a scalar loss depending on many...
autodiff book May 11, 2026
Writes › Book › Auto Diff › Chapter 6. Reverse Mode Automatic Differentiation ›

Case Studies

This section studies reverse mode automatic differentiation through concrete examples. Each case has the same structure:
autodiff book May 11, 2026