Chapter 21. Major AD Systems

ADIFOR, short for Automatic Differentiation of Fortran, is one of the classical source-transformation systems for automatic differentiation. It was designed for numerical...

10 items

Section	Title
1	Chapter 21. Major AD Systems
2	Tapenade
3	TensorFlow Autograd
4	PyTorch Autograd
5	JAX
6	Zygote
7	Enzyme
8	Tinygrad
9	Comparative Architecture Analysis
10	Differentiation of Large Stateful Systems

Writes › Book › Auto Diff › Chapter 21. Major AD Systems ›

Chapter 21. Major AD Systems

ADIFOR, short for Automatic Differentiation of Fortran, is one of the classical source-transformation systems for automatic differentiation. It was designed for numerical...

Writes › Book › Auto Diff › Chapter 21. Major AD Systems ›

Tapenade

Tapenade is a source-transformation automatic differentiation system developed at INRIA. Like ADIFOR, it takes an existing program and produces a new differentiated program....

Writes › Book › Auto Diff › Chapter 21. Major AD Systems ›

TensorFlow Autograd

TensorFlow Autograd refers to TensorFlow’s automatic differentiation system, mainly exposed through tf.GradientTape. It is a reverse-mode AD system designed for tensor...

Writes › Book › Auto Diff › Chapter 21. Major AD Systems ›

PyTorch Autograd

PyTorch Autograd is a dynamic reverse-mode automatic differentiation system. It records tensor operations as they execute, builds a computation graph at runtime, and then...

Writes › Book › Auto Diff › Chapter 21. Major AD Systems ›

JAX

JAX is an automatic differentiation and array programming system for Python. It combines NumPy-like syntax with composable program transformations. Its core transformations...

Writes › Book › Auto Diff › Chapter 21. Major AD Systems ›

Zygote

Zygote is a source-to-source reverse-mode automatic differentiation system for the Julia programming language. It was designed to differentiate high-level Julia code directly,...

Writes › Book › Auto Diff › Chapter 21. Major AD Systems ›

Enzyme

Enzyme is a compiler-based automatic differentiation system for LLVM and MLIR. Instead of differentiating source code directly, or recording tensor operations at runtime,...

Writes › Book › Auto Diff › Chapter 21. Major AD Systems ›

Tinygrad

Tinygrad is a small deep learning framework centered around a minimal reverse-mode automatic differentiation engine. It was created by entity"people","George...

Writes › Book › Auto Diff › Chapter 21. Major AD Systems ›

Comparative Architecture Analysis

The systems in this chapter show that automatic differentiation is not one implementation technique. It is a family of program transformations. Each system chooses a different...

Writes › Book › Auto Diff › Chapter 21. Major AD Systems ›

Differentiation of Large Stateful Systems

Automatic differentiation works naturally on pure mathematical functions: