Chapter 2. Mathematical Foundations

Automatic differentiation begins with a simple object: a function.

8 items

Section	Title
1	Chapter 2. Mathematical Foundations
2	Multivariate Calculus
3	Jacobians and Hessians
4	Chain Rule as Composition Algebra
5	Computational Graphs
6	Linearization
7	Dual Spaces and Pushforwards
8	Taylor Expansions

Writes › Book › Auto Diff › Chapter 2. Mathematical Foundations ›

Chapter 2. Mathematical Foundations

Automatic differentiation begins with a simple object: a function.

Writes › Book › Auto Diff › Chapter 2. Mathematical Foundations ›

Multivariate Calculus

Automatic differentiation is usually applied to functions with many inputs and many outputs. The calculus needed for this setting is multivariate calculus: the study of how a...

Writes › Book › Auto Diff › Chapter 2. Mathematical Foundations ›

Jacobians and Hessians

The gradient is enough when a function has many inputs and one scalar output. More general programs need more general derivative objects. Two of the most important are the...

Writes › Book › Auto Diff › Chapter 2. Mathematical Foundations ›

Chain Rule as Composition Algebra

The chain rule is the central theorem behind automatic differentiation. Every useful AD algorithm is a disciplined way of applying the chain rule to a program.

Writes › Book › Auto Diff › Chapter 2. Mathematical Foundations ›

Computational Graphs

A computational graph represents a calculation as nodes and edges. Nodes represent operations or values. Edges represent data dependencies. Automatic differentiation uses this...

Writes › Book › Auto Diff › Chapter 2. Mathematical Foundations ›

Linearization

Linearization is the operation of replacing a nonlinear function by its best local linear approximation at a chosen point. Automatic differentiation can be understood as a...

Writes › Book › Auto Diff › Chapter 2. Mathematical Foundations ›

Dual Spaces and Pushforwards

Forward mode and reverse mode propagate different kinds of objects.

Writes › Book › Auto Diff › Chapter 2. Mathematical Foundations ›

Taylor Expansions

Differentiation describes how a function changes locally. A Taylor expansion extends this idea by approximating a function with a polynomial around a point.