autodiff | brain

Writes › Book › Auto Diff › Chapter 21. Major AD Systems ›

Differentiation of Large Stateful Systems

Automatic differentiation works naturally on pure mathematical functions: $$ y = f(x). $$ The function receives inputs, computes outputs, and produces no persistent side effects. Most theoretical treatments of AD assume this model. Large real systems rarely behave this way. Operating systems, databases, distributed services, simulations, robotics stacks, game engines, browsers, network services, and long-running machine learning pipelines all contain substantial mutable state. Their behavior depends not only on explicit...

Writes › Book › Auto Diff › Chapter 22. Open Problems ›

Chapter 22. Open Problems

Automatic differentiation works naturally on pure mathematical functions: $$ y = f(x). $$ The function receives inputs, computes outputs, and produces no persistent side effects. Most theoretical treatments of AD assume this model. Large real systems rarely behave this way. Operating systems, databases, distributed services, simulations, robotics stacks, game engines, browsers, network services, and long-running machine learning pipelines all contain substantial mutable state. Their behavior depends not only on explicit...

Writes › Book › Auto Diff › Appendix ›

Differentiation of Large Stateful Systems

Automatic differentiation works naturally on pure mathematical functions: $$ y = f(x). $$ The function receives inputs, computes outputs, and produces no persistent side effects. Most theoretical treatments of AD assume this model. Large real systems rarely behave this way. Operating systems, databases, distributed services, simulations, robotics stacks, game engines, browsers, network services, and long-running machine learning pipelines all contain substantial mutable state. Their behavior depends not only on explicit...

Writes › Book › Auto Diff › Chapter 20. Building an AD Engine ›

Production Deployment

Production Deployment A minimal automatic differentiation engine can compute correct gradients on small programs. A production system must survive long-running workloads, large tensors, distributed execution, hardware failures, user-defined operators, and adversarial numerical conditions. Production deployment transforms AD from a mathematical mechanism into infrastructure. The core derivative rules usually remain small. Most complexity appears in: execution management memory behavior observability reproducibility interoperability fault handling scalability A production AD system is therefore...

Writes › Book › Auto Diff › Chapter 18. Advanced Topics ›

Unified Differentiable Infrastructure

Unified Differentiable Infrastructure Automatic differentiation began as a numerical technique for computing gradients of scalar functions. Modern systems use differentiation far more broadly. A single computation may now include: Component Example neural networks representation learning optimization layers constrained decisions simulators physical dynamics databases retrieval and aggregation probabilistic models uncertainty differential equations continuous dynamics rendering systems graphics and vision distributed systems large-scale training Gradients must propagate through all of them. Unified...

Writes › Book › Auto Diff › Chapter 16. Sparse and Structured Differentiation ›

Chapter 16. Sparse and Structured Differentiation

Sparse and structured differentiation studies how to compute derivatives without materializing dense derivative objects. Many real systems have enormous Jacobians and Hessians, but most entries are zero, repeated, blocked, low rank, or constrained by graph structure. A dense Jacobian for a function $$ f : \mathbb{R}^n \to \mathbb{R}^m $$ has $mn$ entries. If $m$ and $n$ are large, storing or computing the full matrix is often impossible. Automatic differentiation usually...

Writes › Book › Auto Diff › Chapter 13. Optimization and Machine Learning ›

Physics-Informed Models

Physics-Informed Models Physics-informed models combine data fitting with equations from physics or applied mathematics. The model is trained not only to match observed samples, but also to satisfy known laws such as conservation equations, differential equations, boundary conditions, or constitutive relations. A common setting is a neural network approximation to an unknown function: $$ u_\theta(x,t) $$ where $x$ is a spatial coordinate, $t$ is time, and $\theta$ are trainable parameters....

Writes › Book › Auto Diff › Chapter 12. AD in Modern Programming Languages ›

Effect Systems and Mutation

Effect Systems and Mutation Automatic differentiation is easiest to define for pure functions. A pure function behaves like a mathematical mapping: it consumes inputs, produces outputs, and has no observable side effects. Real programs are rarely this simple. They mutate arrays, allocate memory, read state, launch kernels, perform I/O, sample random values, and synchronize across devices. An effect system describes these behaviors explicitly. It tracks what a computation does in...

Writes › Book › Auto Diff › Chapter 11. Compiler and Runtime Design ›

Ahead-of-Time vs Just-in-Time Differentiation

Ahead-of-Time vs Just-in-Time Differentiation Automatic differentiation can be performed before a program runs, while it runs, or in a staged phase between the two. The two main compiler-oriented models are ahead-of-time differentiation and just-in-time differentiation. Ahead-of-time differentiation, or AOT differentiation, generates derivative code before execution. Just-in-time differentiation, or JIT differentiation, generates derivative code during execution, usually after seeing concrete input types, shapes, or runtime configuration. Both models treat AD as...

Writes › Book › Auto Diff › Chapter 6. Reverse Mode Automatic Differentiation ›

Case Studies

Case Studies This section studies reverse mode automatic differentiation through concrete examples. Each case has the same structure: evaluate the primal computation forward; seed the output adjoint; propagate adjoints backward; collect input or parameter gradients. The examples move from scalar expressions to neural network layers and iterative programs. Case Study 1: Scalar Expression Consider $$ y = \log(1 + x^2). $$ Decompose the computation: $$ v_1 = x^2, $$ $$...

Writes › Book › Auto Diff › Chapter 4. Core Theory of Automatic Differentiation ›

Tangent Propagation

Forward mode automatic differentiation computes derivatives by propagating tangent values alongside ordinary values. The ordinary value is called the primal. The derivative value is called the tangent. For every program variable $v$, forward mode tracks: $$ (v,\dot v) $$ The dot notation means “the change in $v$ induced by a chosen change in the input.” If the input changes in direction $\dot x$, then every later value changes according to...

Writes › Book › Auto Diff › Chapter 2. Mathematical Foundations ›

Taylor Expansions

Taylor Expansions Differentiation describes how a function changes locally. A Taylor expansion extends this idea by approximating a function with a polynomial around a point. Automatic differentiation is fundamentally a first-order method in most applications, but higher-order AD is closely connected to Taylor expansions. Forward mode can be generalized to propagate higher-order coefficients, and many derivative identities become clearer through the Taylor viewpoint. Local Polynomial Approximation Let $$ f :...

Writes › Book › Auto Diff › Appendix ›

Comparative Architecture Analysis

Comparative Architecture Analysis The systems in this chapter show that automatic differentiation is not one implementation technique. It is a family of program transformations. Each system chooses a different representation of the program, a different execution model, and a different boundary between user code, compiler code, and runtime code. ADIFOR and Tapenade treat AD as source transformation. TensorFlow and PyTorch treat AD as graph or tape execution over tensor primitives....

Writes › Book › Auto Diff › Chapter 21. Major AD Systems ›

Comparative Architecture Analysis

Comparative Architecture Analysis The systems in this chapter show that automatic differentiation is not one implementation technique. It is a family of program transformations. Each system chooses a different representation of the program, a different execution model, and a different boundary between user code, compiler code, and runtime code. ADIFOR and Tapenade treat AD as source transformation. TensorFlow and PyTorch treat AD as graph or tape execution over tensor primitives....

Writes › Book › Auto Diff › Chapter 20. Building an AD Engine ›

Testing Derivatives

Testing Derivatives An automatic differentiation engine is only useful if its derivatives are correct. A small mistake in a backward rule can silently corrupt optimization, training, or scientific computation. Derivative testing therefore belongs at the center of AD system design. A correct primal computation with incorrect gradients is usually worse than a runtime error. Numerical optimization may still appear to progress while converging to meaningless results. Derivative testing should validate:...

Writes › Book › Auto Diff › Chapter 19. Theory and Foundations ›

Differentiation as Functorial Transformation

Differentiation as Functorial Transformation The preceding sections described automatic differentiation through algebraic, categorical, logical, and denotational models. These viewpoints converge on one central idea: Differentiation preserves structure. A program is built by composing smaller computations. Automatic differentiation works because the derivative of the whole computation can be constructed from derivatives of the parts, in the same compositional order for forward mode and in the dual order for reverse mode. This...

Writes › Book › Auto Diff › Chapter 18. Advanced Topics ›

Verified Differentiation

Verified Differentiation Automatic differentiation systems are usually trusted because they implement mathematically established rules such as the chain rule, product rule, and linearization of elementary operations. However, practical AD systems are large compiler and runtime systems containing: Component Complexity graph transformations compiler correctness tensor kernels numerical precision mutation handling state consistency checkpointing recomputation logic custom gradients user-defined semantics distributed execution synchronization correctness A small mistake may silently produce incorrect gradients....

Writes › Book › Auto Diff › Chapter 17. Numerical and Systems Concerns ›

Distributed Gradient Computation

Distributed Gradient Computation Distributed gradient computation appears when a differentiable program no longer fits comfortably on one device or one machine. The reason may be model size, data volume, sequence length, simulation resolution, or throughput requirements. At that point, automatic differentiation becomes a distributed systems problem. The derivative rules remain local. The execution is no longer local. A distributed AD system must compute gradients while coordinating: partitioned tensors, replicated parameters,...

Writes › Book › Auto Diff › Chapter 15. Differentiable Systems Architecture ›

Summary

Summary Differentiable systems architecture extends automatic differentiation beyond isolated functions and neural network layers. The central idea is to treat larger systems as compositions of differentiable, partially differentiable, or optimization-aware components. A system may include: Component Differentiable Role Pipeline Propagates task loss through many stages Database Makes retrieval, ranking, and aggregation trainable Renderer Sends image loss back into geometry, pose, lighting, and material parameters Physics engine Sends trajectory loss back...

Writes › Book › Auto Diff › Chapter 14. Scientific Computing Applications ›

Probabilistic Programming

Probabilistic Programming Probabilistic programming represents uncertainty using executable probabilistic models. A probabilistic program defines a distribution rather than only a deterministic computation. Automatic differentiation is important because most modern probabilistic inference methods rely on gradients of probability densities, likelihoods, or variational objectives. A probabilistic model typically defines: $$ x \sim p(x \mid \theta), $$ where: Symbol Meaning $x$ latent variables $\theta$ model parameters $p$ probability distribution Given observations $y$, inference...

Writes › Book › Auto Diff › Chapter 13. Optimization and Machine Learning ›

Reinforcement Learning

Reinforcement Learning Reinforcement learning studies learning systems that act in an environment. Unlike supervised learning, the training signal is not a target label for each input. The model chooses actions, the environment responds, and rewards may arrive later. A reinforcement learning problem is usually described as a Markov decision process. At time $t$, the agent observes a state $s_t$, chooses an action $a_t$, receives a reward $r_t$, and moves to...

Writes › Book › Auto Diff › Chapter 12. AD in Modern Programming Languages ›

Type Systems for Differentiation

Type Systems for Differentiation Automatic differentiation interacts deeply with type systems because differentiation changes the structure of computation. A derivative operator maps one function into another function with different inputs, outputs, and intermediate behavior. A type system can describe these transformations explicitly, reject invalid programs, and encode mathematical structure directly into the language. In small AD systems, differentiation is often treated as a runtime feature. In larger differentiable programming systems,...

Writes › Book › Auto Diff › Chapter 11. Compiler and Runtime Design ›

Kernel Fusion

Kernel Fusion Kernel fusion combines several small operations into one larger executable unit. In tensor programs, a kernel is a unit of work launched on a device such as a CPU, GPU, or TPU. A simple expression may produce many kernels if each primitive operation is executed separately. y = exp(x) z = y + 1 w = log(z) A naive runtime may execute three kernels: kernel 1: y =...

Writes › Book › Auto Diff › Chapter 10. Matrix and Tensor Differentiation ›

GPU Tensor Kernels

GPU Tensor Kernels Modern automatic differentiation systems are fundamentally tensor compiler systems. Their performance depends less on mathematical differentiation rules than on how efficiently tensor kernels execute on accelerators. GPU tensor kernels are the computational substrate of modern deep learning, scientific computing, and differentiable simulation. Reverse-mode AD amplifies the importance of kernel efficiency because every forward operation usually induces at least one backward operation. The challenge is therefore not only:...

Writes › Book › Auto Diff › Chapter 8. Higher-Order Differentiation ›

Complexity of Higher Orders

Complexity of Higher Orders Higher-order automatic differentiation faces a fundamental problem: derivative structure grows combinatorially with order. For a function $$ f : \mathbb{R}^n \to \mathbb{R}, $$ the first derivative is a vector, the second derivative is a matrix, and higher derivatives become tensors. The size growth is severe: Order Object Dense size 1 gradient $n$ 2 Hessian $n^2$ 3 third-order tensor $n^3$ 4 fourth-order tensor $n^4$ This growth is...

Writes › Book › Auto Diff › Chapter 7. Dual Numbers and Algebraic Structures ›

Differential Lambda Calculus

Differential Lambda Calculus Automatic differentiation is deeply connected to functional programming and lambda calculus. Programs can be viewed as mathematical functions, and differentiation can be treated as a transformation on programs themselves. Differential lambda calculus extends ordinary lambda calculus with explicit notions of infinitesimal variation, linear approximation, and derivative propagation. This provides a formal foundation for differentiable programming languages. Ordinary Lambda Calculus Lambda calculus is a minimal formal system for...

Writes › Book › Auto Diff › Chapter 6. Reverse Mode Automatic Differentiation ›

Reverse Mode in Deep Learning

Reverse Mode in Deep Learning Reverse mode automatic differentiation is the mathematical and systems basis of backpropagation. In deep learning, the objective is usually a scalar loss depending on many parameters. This is exactly the setting where reverse mode is most efficient. A neural network training step can be written as $$ L(\theta) = \ell(F(x;\theta), y), $$ where $x$ is an input batch, $y$ is the target, $F$ is the...

Writes › Book › Auto Diff › Chapter 5. Forward Mode Automatic Differentiation ›

Case Studies

Case Studies Forward mode automatic differentiation appears in many numerical systems where directional derivatives, local sensitivities, or small parameter sets are important. This chapter examines concrete applications and analyzes why forward mode fits their computational structure. The emphasis is not merely on using derivatives, but on how tangent propagation integrates with the execution model of each system. Sensitivity Analysis in Scientific Computing Many scientific simulations depend on a small set...

Writes › Book › Auto Diff › Chapter 4. Core Theory of Automatic Differentiation ›

Summary and Synthesis

Automatic differentiation is a method for computing derivatives by transforming programs into derivative-propagating computations. It does not approximate derivatives numerically and does not manipulate symbolic expressions globally. Instead, it applies the chain rule locally across primitive operations. This chapter established the operational foundations of AD: local derivative propagation elementary derivative rules forward accumulation reverse accumulation mixed-mode composition computational complexity floating point exactness program transformation Together, these ideas define the modern...

Writes › Book › Auto Diff › Appendix ›

Tinygrad

Tinygrad Tinygrad is a small deep learning framework centered around a minimal reverse-mode automatic differentiation engine. It was created by entity["people","George Hotz","Tinygrad creator"] as an experiment in reducing machine learning infrastructure to a compact and understandable core. Unlike large frameworks such as entity["company","Google","Mountain View, CA, USA"] TensorFlow or entity["company","Meta","Menlo Park, CA, USA"] PyTorch, Tinygrad emphasizes simplicity over ecosystem breadth. Its importance is educational and architectural rather than industrial scale. Tinygrad...

Writes › Book › Auto Diff › Chapter 21. Major AD Systems ›

Tinygrad

Tinygrad Tinygrad is a small deep learning framework centered around a minimal reverse-mode automatic differentiation engine. It was created by entity["people","George Hotz","Tinygrad creator"] as an experiment in reducing machine learning infrastructure to a compact and understandable core. Unlike large frameworks such as entity["company","Google","Mountain View, CA, USA"] TensorFlow or entity["company","Meta","Menlo Park, CA, USA"] PyTorch, Tinygrad emphasizes simplicity over ecosystem breadth. Its importance is educational and architectural rather than industrial scale. Tinygrad...

Writes › Book › Auto Diff › Chapter 20. Building an AD Engine ›

Performance Benchmarking

Performance Benchmarking Performance benchmarking measures whether an automatic differentiation engine is fast, memory-efficient, and scalable under realistic workloads. It also protects the engine from regressions. Without benchmarks, small implementation changes can silently increase allocation count, tape size, kernel launch count, or backward latency. A benchmark should answer concrete questions: How fast is the forward pass? How fast is the backward pass? How much memory is retained? How many allocations occur?...

Writes › Book › Auto Diff › Chapter 19. Theory and Foundations ›

Denotational Models

Denotational Models Operational semantics explains how automatic differentiation executes. Denotational semantics explains what differentiable programs mean. A denotational model assigns mathematical objects to programs: numbers, functions, spaces, morphisms, domains, linear maps. The meaning of a program becomes independent of: execution order, implementation strategy, compiler structure, hardware behavior. For automatic differentiation, denotational semantics answers a central question: What mathematical object does a differentiable program denote? This question becomes increasingly important for:...

Writes › Book › Auto Diff › Chapter 18. Advanced Topics ›

Differentiable Programming Languages

Differentiable Programming Languages Automatic differentiation began as a transformation applied to numerical programs. A differentiable programming language instead treats differentiation as a native semantic operation of the language itself. In such systems, derivatives are not external utilities layered on top of programs. They become part of the programming model. The language may support constructs such as: grad(f) jacobian(f) vjp(f) jvp(f) as ordinary language operators. The goal is deeper integration between:...

Writes › Book › Auto Diff › Chapter 17. Numerical and Systems Concerns ›

GPU and TPU Execution

GPU and TPU Execution Modern automatic differentiation systems are built around accelerator hardware. GPUs and TPUs provide enormous throughput for tensor operations, making large-scale differentiable computation practical. Reverse-mode AD, deep learning, differentiable simulation, and large tensor algebra all depend heavily on these architectures. Accelerators are not merely faster CPUs. They expose fundamentally different execution models: massive parallelism, hierarchical memory systems, SIMD or SIMT execution, high-throughput tensor units, asynchronous scheduling, and...

Writes › Book › Auto Diff › Chapter 15. Differentiable Systems Architecture ›

Hybrid Symbolic-Numeric Systems

Hybrid Symbolic-Numeric Systems A hybrid symbolic-numeric system combines discrete symbolic reasoning with continuous numerical computation. In the context of automatic differentiation, it means a pipeline where some components operate over symbols, rules, logic, programs, graphs, or queries, while other components operate over tensors, vectors, probabilities, or differentiable functions. The basic form is: $$ x \rightarrow S(x) \rightarrow N(S(x)) \rightarrow L $$ where $S$ is a symbolic transformation, $N$ is a...

Writes › Book › Auto Diff › Chapter 14. Scientific Computing Applications ›

Robotics and Control

Robotics and Control Robotics and control systems interact with the physical world through sensing, estimation, planning, and actuation. Automatic differentiation is important because modern control pipelines increasingly depend on optimization, simulation, system identification, and differentiable models. A robotic system is often represented as a dynamical system: $$ x_{t+1}=f(x_t,u_t,\theta), $$ where: Symbol Meaning $x_t$ system state $u_t$ control input $\theta$ physical or model parameters A controller chooses actions $$ u_t =...

Writes › Book › Auto Diff › Chapter 13. Optimization and Machine Learning ›

Meta-Learning

Meta-Learning Meta-learning studies systems that improve how they learn. Instead of only optimizing model parameters for one task, a meta-learning method optimizes some part of the learning process itself: an initialization, an optimizer, a loss function, a data-selection rule, a representation, or an adaptation procedure. A standard supervised learning problem learns parameters $\theta$ for one dataset. A meta-learning problem considers a distribution over tasks: $$ \tau \sim p(\tau). $$ Each...

Writes › Book › Auto Diff › Chapter 12. AD in Modern Programming Languages ›

AD in Swift

AD in Swift Swift became an important experiment in language-integrated automatic differentiation because it attempted to make differentiation a core compiler feature rather than a library layered on top of the language. The central idea was that differentiable functions should participate in the type system, compilation pipeline, optimization passes, and language semantics directly. This differs from many runtime-based systems. Instead of tracing operations dynamically or overloading tensor types externally, the...

Writes › Book › Auto Diff › Chapter 11. Compiler and Runtime Design ›

Memory Planning

Memory Planning Memory planning determines where values are stored, how long they remain alive, and when storage can be reused. For automatic differentiation, memory planning is a central systems problem because reverse mode often requires large numbers of intermediate values from the forward pass. A simple mathematical expression may appear small: y = sin(x * x) but a deep neural network, PDE solver, or simulation may produce millions of intermediate...

Writes › Book › Auto Diff › Chapter 10. Matrix and Tensor Differentiation ›

Sparse Tensor Derivatives

Sparse Tensor Derivatives Most real computational problems are sparse. Large matrices and tensors often contain mostly zeros, structured blocks, or local interactions. Sparse representations reduce memory usage and computational cost by storing only the nonzero structure. Automatic differentiation must preserve sparsity whenever possible. If differentiation destroys sparsity structure, the resulting gradients can become prohibitively expensive. Sparse differentiation therefore has two goals: Compute correct derivatives. Preserve structural sparsity through the derivative...

Writes › Book › Auto Diff › Chapter 9. Differentiation of Control Flow ›

Exception Handling and Undefined Regions

Exception Handling and Undefined Regions Programs do not only branch between valid computations. They also fail, stop early, raise exceptions, return sentinel values, or enter undefined numerical regions. These behaviors matter for automatic differentiation because a derivative only has meaning where the primal computation itself has a well-defined value. A mathematical function may be partial: $$ f : D \to Y $$ where $D$ is the subset of inputs for...

Writes › Book › Auto Diff › Chapter 8. Higher-Order Differentiation ›

Perturbation Confusion

Perturbation Confusion Perturbation confusion is a correctness bug that appears in nested automatic differentiation, especially nested forward mode. It happens when two derivative computations accidentally use the same infinitesimal perturbation. Forward mode is often explained with dual numbers: $$ x + \epsilon \dot{x}, $$ where $$ \epsilon^2 = 0. $$ The symbol $\epsilon$ marks the tangent part of the computation. In a single forward-mode pass, this works cleanly. In nested...

Writes › Book › Auto Diff › Chapter 7. Dual Numbers and Algebraic Structures ›

Differential Lambda Calculus

Differential Lambda Calculus Automatic differentiation is deeply connected to functional programming and lambda calculus. Programs can be viewed as mathematical functions, and differentiation can be treated as a transformation on programs themselves. Differential lambda calculus extends ordinary lambda calculus with explicit notions of infinitesimal variation, linear approximation, and derivative propagation. This provides a formal foundation for differentiable programming languages. Ordinary Lambda Calculus Lambda calculus is a minimal formal system for...

Writes › Book › Auto Diff › Chapter 6. Reverse Mode Automatic Differentiation ›

Checkpointing

Checkpointing Checkpointing is a technique for reducing the memory cost of reverse mode automatic differentiation by selectively storing intermediate states and recomputing missing values during the backward pass. It addresses the central problem of reverse mode: The backward pass needs information from the forward pass, but storing all intermediate values may exceed available memory. Checkpointing balances memory usage against recomputation cost. Basic Idea Suppose a forward computation produces a sequence...

Writes › Book › Auto Diff › Chapter 5. Forward Mode Automatic Differentiation ›

Sparse Forward Methods

Sparse Forward Methods Many real-world Jacobians are sparse. Most derivative entries are zero because outputs depend only on small subsets of inputs. Forward mode automatic differentiation can exploit this structure to reduce: runtime, memory usage, tangent dimensionality, cache pressure, bandwidth cost. Sparse forward methods are techniques for propagating only the derivative information that actually matters. Sparse Jacobians Suppose $$ f : \mathbb{R}^n \to \mathbb{R}^m. $$ The Jacobian is $$ J_f(x)...

Writes › Book › Auto Diff › Chapter 4. Core Theory of Automatic Differentiation ›

AD as Program Transformation

Automatic differentiation can be understood as a transformation from one program into another program. The original program computes values. The transformed program computes values and derivatives. This view is important because real AD systems operate on code, traces, graphs, or compiler intermediate representations. They are not merely applying calculus rules on paper. They rewrite computation. From Mathematical Function to Program A mathematical function may be written as: $$ f(x)=\sin(x^2+3x) $$...

Writes › Book › Auto Diff › Chapter 3. Programs as Mathematical Objects ›

Differentiable Subprograms

Differentiable Subprograms A differentiable subprogram is a program fragment that can participate in derivative propagation as a coherent unit. Instead of differentiating an entire application monolithically, AD systems decompose computation into smaller callable pieces with well-defined derivative behavior. A simple example is: def square(x): return x * x Mathematically: $$ f(x) = x^2. $$ Its derivative is: $$ f'(x) = 2x. $$ An AD system can either inline the body...

Writes › Book › Auto Diff › Appendix ›

Enzyme

Enzyme Enzyme is a compiler-based automatic differentiation system for LLVM and MLIR. Instead of differentiating source code directly, or recording tensor operations at runtime, Enzyme works on compiler intermediate representation. This design gives Enzyme a distinct position among AD systems. ADIFOR and Tapenade transform source programs. PyTorch records a dynamic tape. JAX traces array programs. Zygote transforms Julia IR. Enzyme differentiates low-level optimized IR produced by many frontends. The project...

Writes › Book › Auto Diff › Chapter 21. Major AD Systems ›

Enzyme

Enzyme Enzyme is a compiler-based automatic differentiation system for LLVM and MLIR. Instead of differentiating source code directly, or recording tensor operations at runtime, Enzyme works on compiler intermediate representation. This design gives Enzyme a distinct position among AD systems. ADIFOR and Tapenade transform source programs. PyTorch records a dynamic tape. JAX traces array programs. Zygote transforms Julia IR. Enzyme differentiates low-level optimized IR produced by many frontends. The project...

Writes › Book › Auto Diff › Chapter 20. Building an AD Engine ›

Custom Gradients

Custom Gradients A custom gradient gives the user direct control over the backward rule of an operation. The forward computation still produces an ordinary value, but the derivative no longer has to be inferred from the primitive operations used internally. This is useful when the default AD result is mathematically correct but computationally poor, numerically unstable, too memory-hungry, or simply not the derivative the user wants the system to use....

Writes › Book › Auto Diff › Chapter 19. Theory and Foundations ›

Formal Verification

Formal Verification Automatic differentiation systems are trusted infrastructure. Scientific computing, machine learning, optimization, simulation, and control systems depend on gradients being correct. Yet AD implementations are complicated: compiler transformations, graph rewrites, mutation handling, checkpointing, floating-point arithmetic, higher-order differentiation, distributed execution. A small semantic error may silently produce incorrect gradients. Formal verification studies how to prove that: differentiation rules are correct, transformed programs preserve semantics, gradients satisfy mathematical definitions, AD compilers...

Writes › Book › Auto Diff › Chapter 18. Advanced Topics ›

Quantum Differentiation

Quantum Differentiation Quantum computation introduces a computational model fundamentally different from classical programs. A classical program evolves deterministic or probabilistic states through ordinary arithmetic and control flow. A quantum program evolves complex-valued amplitudes through unitary transformations and measurement operators. Automatic differentiation in quantum systems studies how outputs of quantum computations change with respect to parameters. This includes gradients of expectation values, variational quantum circuits, quantum control systems, and hybrid quantum-classical...

Writes › Book › Auto Diff › Chapter 17. Numerical and Systems Concerns ›

Parallelism

Parallelism Automatic differentiation is usually described as a transformation of programs or computational graphs. In real systems, it is also a parallel execution problem. Large differentiable programs contain millions or billions of operations, and modern hardware exposes thousands of concurrent execution lanes. Efficient AD must therefore expose, schedule, and control parallel work. Parallelism affects both performance and numerical behavior. It changes operation ordering, memory movement, reduction structure, and sometimes even...

Writes › Book › Auto Diff › Chapter 15. Differentiable Systems Architecture ›

Differentiable Operating Systems

Differentiable Operating Systems A differentiable operating system is an execution environment whose resource-management decisions can be optimized using gradients or gradient-like feedback. Instead of treating scheduling, memory placement, caching, and I/O as fixed policies, the system exposes parts of those policies as trainable components. The goal is not to replace an operating system kernel with a neural network. The goal is to make selected system decisions measurable, optimizable, and responsive...

Writes › Book › Auto Diff › Chapter 14. Scientific Computing Applications ›

Signal Processing

Signal Processing Signal processing studies how information is represented, transformed, filtered, compressed, reconstructed, and estimated from signals. A signal may be a time series, an image, an audio waveform, a radar return, a sensor stream, or a multidimensional field. Automatic differentiation is useful because many signal processing systems can be written as differentiable programs: $$ x \to T_\theta(x) \to y \to L(y) $$ where $x$ is an input signal, $T_\theta$...

Writes › Book › Auto Diff › Chapter 13. Optimization and Machine Learning ›

Implicit Layers

Implicit Layers An implicit layer defines its output as the solution of an equation, not as a fixed sequence of explicit operations. Instead of computing $$ z = f_\theta(x) $$ by applying a known list of layers, an implicit layer defines $z$ through a condition: $$ F(z, x, \theta) = 0. $$ The value $z$ is found by a solver. The solver may use Newton iterations, fixed-point iteration, conjugate gradients,...

Writes › Book › Auto Diff › Chapter 12. AD in Modern Programming Languages ›

AD in Julia

AD in Julia Julia was designed for high-performance technical computing. It combines interactive syntax with a compiler capable of specializing code aggressively based on types. This makes it a strong environment for automatic differentiation because AD systems can operate close to ordinary mathematical code while still generating optimized machine code. Unlike Python, where tensor operations are often dispatched into external runtimes, Julia programs are commonly written directly in the host...

Writes › Book › Auto Diff › Chapter 11. Compiler and Runtime Design ›

Staging and Partial Evaluation

Staging and Partial Evaluation Staging is the separation of a program into phases. In an automatic differentiation system, the most common phases are: trace time compile time run time A staged system does not execute every part of the user program in the same way. Some parts run immediately in the host language. Other parts are captured into an intermediate representation. That captured representation may then be differentiated, optimized, compiled,...

Writes › Book › Auto Diff › Chapter 10. Matrix and Tensor Differentiation ›

Singular Value Decomposition

Singular Value Decomposition The singular value decomposition (SVD) is one of the most important matrix factorizations in numerical linear algebra. It appears in dimensionality reduction, least squares, low-rank approximation, spectral regularization, recommendation systems, scientific computing, and modern machine learning. For automatic differentiation, SVD is both powerful and difficult. Singular values are comparatively stable. Singular vectors are not. The derivative formulas expose this instability directly. Definition For a matrix $$ A...

Writes › Book › Auto Diff › Chapter 9. Differentiation of Control Flow ›

Differentiating Stateful Systems

Differentiating Stateful Systems A stateful system is a program whose output depends not only on its explicit inputs, but also on stored state. The state may live in variables, objects, arrays, files, random number generators, caches, mutable buffers, optimizers, simulators, databases, or external services. In pure mathematical notation, a function has the form: $$ y = f(x) $$ A stateful computation has the form: $$ (y, s') = F(x, s)...

Writes › Book › Auto Diff › Chapter 8. Higher-Order Differentiation ›

Efficient Higher-Order Methods

Efficient Higher-Order Methods Higher-order derivatives contain rich geometric information, but naïve computation quickly becomes impractical. For a function $$ f : \mathbb{R}^n \to \mathbb{R}, $$ the size of derivative objects grows rapidly: Derivative Size gradient $n$ Hessian $n^2$ third-order tensor $n^3$ fourth-order tensor $n^4$ Even when computation is mathematically possible, explicit tensor construction is often infeasible. Efficient higher-order methods therefore focus on three goals: Goal Strategy avoid explicit tensors operator...

Writes › Book › Auto Diff › Chapter 7. Dual Numbers and Algebraic Structures ›

Category-Theoretic View

Category-Theoretic View Automatic differentiation can be described operationally through dual numbers and computational graphs. It can also be described abstractly using category theory. This perspective treats differentiation as a structural transformation between compositional systems. Instead of focusing on coordinates, variables, or symbolic expressions, category theory focuses on: objects morphisms composition functorial structure The central idea is that differentiation preserves composition. Automatic differentiation becomes a compositional lifting process. Categories A category...

Writes › Book › Auto Diff › Chapter 6. Reverse Mode Automatic Differentiation ›

Memory-Time Tradeoffs

Memory-Time Tradeoffs Reverse mode automatic differentiation is computationally efficient for scalar-output functions, but it has a major systems cost: it needs information from the forward pass during the backward pass. The reverse pass cannot usually compute derivatives from the output value alone. It needs intermediate primal values, shapes, branches, and sometimes full solver or kernel state. The system must either store this information or recompute it. This creates the central...

Writes › Book › Auto Diff › Chapter 5. Forward Mode Automatic Differentiation ›

Efficient Seeding Strategies

Efficient Seeding Strategies Forward mode automatic differentiation computes Jacobian-vector products: $$ J_f(x)v. $$ The seed vector $$ v $$ determines which derivative information propagates through the program. Choosing good seeds is therefore central to efficiency. A naive approach computes one Jacobian column at a time using basis vectors: $$ e_1, e_2, \ldots, e_n. $$ This always works, but it may waste computation when the Jacobian has structure. Efficient seeding strategies...

Writes › Book › Auto Diff › Chapter 4. Core Theory of Automatic Differentiation ›

Numerical Exactness up to Floating Point

Automatic differentiation computes derivatives exactly with respect to the executed floating point program. This distinguishes AD from numerical differentiation, which approximates derivatives using perturbations. The phrase “exact up to floating point” has a precise meaning: AD does not introduce truncation error AD does not approximate derivatives by finite differences AD propagates analytically correct local derivatives all remaining error comes from ordinary floating point arithmetic Automatic differentiation therefore inherits the numerical...

Writes › Book › Auto Diff › Chapter 3. Programs as Mathematical Objects ›

Purity and Side Effects

Purity and Side Effects A pure computation is easier to differentiate because every output is determined only by its explicit inputs. There is no hidden state, no external mutation, and no dependence on execution history outside the function call. For example: y = f(x) is pure when the same x always produces the same y , and the call changes nothing outside itself. An impure computation may read or write...

Writes › Book › Auto Diff › Chapter 2. Mathematical Foundations ›

Dual Spaces and Pushforwards

Dual Spaces and Pushforwards Forward mode and reverse mode propagate different kinds of objects. Forward mode propagates tangent vectors. These represent small changes in values. Reverse mode propagates cotangent vectors. These represent sensitivities of an output with respect to values. The distinction is more than notation. It explains why forward mode computes Jacobian-vector products, while reverse mode computes vector-Jacobian products. Tangent Vectors Suppose $$ f : \mathbb{R}^n \to \mathbb{R}^m $$...

Writes › Book › Auto Diff › Chapter 1. Introduction ›

Applications Across Science and Engineering

Applications Across Science and Engineering Automatic differentiation became important because derivatives are required everywhere numerical models are optimized, controlled, calibrated, or analyzed. Once a system can compute derivatives automatically, entire classes of algorithms become practical at large scale. The same mathematical mechanism appears across many fields: $$ \text{compute value} \quad \rightarrow \quad \text{compute sensitivity}. $$ The value may be a loss, simulation result, likelihood, energy, trajectory, or prediction. The derivative...

Writes › Book › Auto Diff › Appendix ›

Zygote

Zygote Zygote is a source-to-source reverse-mode automatic differentiation system for the Julia programming language. It was designed to differentiate high-level Julia code directly, without requiring a separate graph-building runtime or explicit tracing API. Its central idea is that differentiation should operate on ordinary language semantics rather than on a restricted tensor graph. A Julia function is treated as a differentiable program transformation target. Differentiating Julia Functions In Zygote, gradients are...

Writes › Book › Auto Diff › Chapter 21. Major AD Systems ›

Zygote

Zygote Zygote is a source-to-source reverse-mode automatic differentiation system for the Julia programming language. It was designed to differentiate high-level Julia code directly, without requiring a separate graph-building runtime or explicit tracing API. Its central idea is that differentiation should operate on ordinary language semantics rather than on a restricted tensor graph. A Julia function is treated as a differentiable program transformation target. Differentiating Julia Functions In Zygote, gradients are...

Writes › Book › Auto Diff › Chapter 20. Building an AD Engine ›

Operator Libraries

Operator Libraries An automatic differentiation engine becomes useful only after it supports a sufficiently rich set of primitive operations. The collection of these primitives is the operator library. The operator library defines: which computations are differentiable how derivatives propagate what tensor semantics exist how shapes and dtypes behave which kernels execute on which devices In a minimal engine, the operator library may contain fewer than ten scalar operations. In a...

Writes › Book › Auto Diff › Chapter 19. Theory and Foundations ›

Program Equivalence

Program Equivalence Automatic differentiation transforms programs. A fundamental semantic question therefore arises: When do two programs have the same derivative behavior? Program equivalence studies when two computations should be considered identical. In differentiable systems this becomes more subtle because: two programs may compute the same values but different derivatives, two programs may produce equivalent derivatives through different computational structures, floating-point behavior may preserve primal equivalence while changing gradient behavior, mutation...

Writes › Book › Auto Diff › Chapter 18. Advanced Topics ›

Probabilistic Automatic Differentiation

Probabilistic Automatic Differentiation Classical automatic differentiation computes derivatives of deterministic programs. A probabilistic program instead describes random variables, probability distributions, and stochastic transformations. The output is not a single value but a distribution, expectation, likelihood, or sampled trajectory. Probabilistic automatic differentiation studies how derivatives propagate through such systems. This includes: Problem Example differentiating expectations stochastic optimization differentiating sampling procedures variational inference differentiating probabilistic programs Bayesian learning differentiating Monte Carlo estimators...

Writes › Book › Auto Diff › Chapter 17. Numerical and Systems Concerns ›

Determinism and Reproducibility

Determinism and Reproducibility Automatic differentiation systems are often assumed to be deterministic. Given identical inputs, identical parameters, and identical code, many users expect identical gradients and identical optimization trajectories. In practice, this assumption frequently fails. Modern differentiable systems execute across: massively parallel hardware, distributed accelerators, asynchronous communication networks, mixed precision arithmetic, optimized compiler pipelines, stochastic training procedures. Small numerical differences accumulate over millions or billions of operations. Two executions of...

Writes › Book › Auto Diff › Chapter 15. Differentiable Systems Architecture ›

Differentiable Compilers

Differentiable Compilers A differentiable compiler is a compilation system that supports gradient propagation through compilation decisions, generated programs, or execution behavior. Instead of treating compilation as a purely symbolic transformation, the compiler becomes part of an optimization process. The core structure is: $$ P \mapsto C(P; \theta) \mapsto E \mapsto L $$ where: Symbol Meaning $P$ Program or computation graph $C$ Compiler transformation $\theta$ Compiler parameters or policies $E$ Executed...

Writes › Book › Auto Diff › Chapter 14. Scientific Computing Applications ›

Computational Finance

Computational Finance Computational finance uses numerical models to price contracts, measure risk, and optimize portfolios. Automatic differentiation is useful because most financial computations depend on many inputs: interest rates, volatilities, correlations, prices, credit spreads, liquidity parameters, and model calibration variables. The central object is usually a value function $$ V = F(x,\theta), $$ where $x$ is market data and $\theta$ is a model parameter vector. The derivative $$ \frac{\partial V}{\partial...

Writes › Book › Auto Diff › Chapter 13. Optimization and Machine Learning ›

Attention Mechanisms

Attention Mechanisms Attention is a sequence operation that lets each position read information from other positions. Instead of compressing the whole past into one recurrent hidden state, attention builds a context-dependent representation by comparing queries with keys and mixing values. Given an input sequence represented as a matrix $$ X \in \mathbb{R}^{T \times d_{\text{model}}}, $$ we construct three projections: $$ Q = XW_Q, \qquad K = XW_K, \qquad V =...

Writes › Book › Auto Diff › Chapter 12. AD in Modern Programming Languages ›

AD in Rust

AD in Rust Rust is an attractive language for automatic differentiation because it combines low-level performance with strong static guarantees. It gives the programmer control over memory layout and allocation, while the type system tracks ownership, borrowing, lifetimes, mutation, and thread safety. These properties are useful for building AD systems that are fast, safe, and explicit about state. Rust also makes AD harder in some places. Operator overloading is available...

Writes › Book › Auto Diff › Chapter 11. Compiler and Runtime Design ›

Tracing Systems

Tracing Systems Tracing is an implementation strategy where an AD system observes a program while it runs and records the operations that occur. The result of tracing is a trace: a simplified program, tape, graph, or intermediate representation that represents one execution of the original program. A normal function may look like this: def f(x): y = x * x z = sin(y) return z When executed with trace-aware values,...

Writes › Book › Auto Diff › Chapter 10. Matrix and Tensor Differentiation ›

Eigenvalue Problems

Eigenvalue Problems Eigenvalue problems are fundamental in numerical analysis, optimization, physics, graph methods, control theory, and machine learning. They are also among the most subtle operations in automatic differentiation. The main difficulty is that eigenvectors are not stable objects. Small perturbations in the matrix can produce large changes in eigenvectors when eigenvalues are close or repeated. The derivative formulas reveal this instability directly. Eigenvalues and Eigenvectors For a square matrix...

Writes › Book › Auto Diff › Chapter 9. Differentiation of Control Flow ›

Non-Smooth Programs

Non-Smooth Programs A non-smooth program contains operations where the derivative is undefined, discontinuous, set-valued, or unstable under small perturbations. These programs arise naturally in optimization, machine learning, geometry, combinatorics, simulation, and systems code. Examples include: absolute value maximum and minimum clipping sorting thresholding indexing quantization branching discrete sampling collision detection contact mechanics Automatic differentiation can still execute on such programs, but the returned gradients require careful interpretation. Smoothness and Classical...

Writes › Book › Auto Diff › Chapter 8. Higher-Order Differentiation ›

Taylor Mode AD

Taylor Mode AD Taylor mode automatic differentiation computes derivatives by propagating truncated Taylor series through a program. Instead of tracking only values and first derivatives, Taylor mode tracks entire local polynomial expansions. For a scalar function $$ f(x), $$ expanded around point $x_0$, $$ f(x_0 + h) = f(x_0) + f'(x_0)h + \frac{f''(x_0)}{2!}h^2 + \frac{f^{(3)}(x_0)}{3!}h^3 + \cdots $$ Taylor mode computes these coefficients automatically. This gives direct access to higher-order...

Writes › Book › Auto Diff › Chapter 7. Dual Numbers and Algebraic Structures ›

Differential Algebras

Differential Algebras Dual numbers and hyper-dual numbers are special cases of a broader algebraic structure called a differential algebra. This framework abstracts differentiation away from specific coordinate formulas and treats derivatives as algebraic operators acting on computational expressions. Automatic differentiation can be understood as evaluation inside differential algebras. This viewpoint unifies: dual numbers truncated polynomial systems higher-order differentiation symbolic differentiation tangent propagation under a common algebraic structure. Algebra and Differentiation...

Writes › Book › Auto Diff › Chapter 6. Reverse Mode Automatic Differentiation ›

Wengert Lists

Wengert Lists A Wengert list is a linear representation of a computation in which every intermediate result is assigned to a unique variable. It is one of the earliest and most influential formulations of automatic differentiation. The central idea is simple: decompose a computation into elementary operations; assign each intermediate result a new identifier; record operations in execution order. This transforms a program into a sequence of explicit data dependencies....

Writes › Book › Auto Diff › Chapter 5. Forward Mode Automatic Differentiation ›

Higher-Dimensional Tangent Spaces

Higher-Dimensional Tangent Spaces So far, forward mode has propagated a single tangent direction: $$ x \mapsto (x, \dot{x}), $$ where $$ \dot{x} \in \mathbb{R}. $$ This computes one directional derivative: $$ J_f(x)v. $$ However, many applications require several directional derivatives simultaneously. Instead of propagating one tangent scalar, we can propagate an entire tangent vector space. Each variable becomes $$ x \mapsto (x, \dot{x}), \qquad \dot{x} \in \mathbb{R}^k. $$ Now every...

Writes › Book › Auto Diff › Chapter 4. Core Theory of Automatic Differentiation ›

Computational Complexity

Automatic differentiation is fundamentally a computational technique. Its practical importance comes from the fact that derivatives can often be computed with asymptotic cost close to the original function evaluation. This section studies the computational complexity of automatic differentiation: operation count memory usage asymptotic scaling dimensional dependence graph structure effects The central result is that derivative computation is usually much cheaper than symbolic differentiation and much more accurate than numerical differentiation....

Writes › Book › Auto Diff › Chapter 3. Programs as Mathematical Objects ›

Memory and State

Memory and State Automatic differentiation operates on computations, but computations execute inside a memory model. Variables occupy storage locations, arrays are mutated, buffers are reused, and values may outlive the operations that created them. Differentiation therefore depends not only on algebraic structure, but also on program state. A simple mathematical expression: $$ y = x^2 + 1 $$ contains no explicit memory model. A program implementation does. tmp = x...

Writes › Book › Auto Diff › Chapter 2. Mathematical Foundations ›

Linearization

Linearization Linearization is the operation of replacing a nonlinear function by its best local linear approximation at a chosen point. Automatic differentiation can be understood as a machine for computing this local linear approximation, or products involving it, for programs. Let $$ f : \mathbb{R}^n \to \mathbb{R}^m $$ At a point $x$, the value of the function is $$ y = f(x) $$ If the input is perturbed by a...

Writes › Book › Auto Diff › Chapter 1. Introduction ›

Historical Development

Historical Development Automatic differentiation developed from a simple observation: a numerical computation already contains the structure needed to compute its derivative. The program evaluates elementary operations in a definite order. If each elementary operation has a derivative rule, then the chain rule can be applied along the same order, either forward or backward. This idea appeared before modern machine learning, before tensor libraries, and before contemporary compiler infrastructure. Its roots...

Writes › Book › Auto Diff › Chapter 21. Major AD Systems ›

JAX

JAX JAX is an automatic differentiation and array programming system for Python. It combines NumPy-like syntax with composable program transformations. Its core transformations include differentiation, vectorization, just-in-time compilation, and parallelization. The important design point is that JAX treats numerical Python functions as transformable objects. A function can be differentiated with grad , batched with vmap , compiled with jit , and distributed with pmap or related sharding APIs. Functional Numerical...

Writes › Book › Auto Diff › Appendix ›

JAX

JAX JAX is an automatic differentiation and array programming system for Python. It combines NumPy-like syntax with composable program transformations. Its core transformations include differentiation, vectorization, just-in-time compilation, and parallelization. The important design point is that JAX treats numerical Python functions as transformable objects. A function can be differentiated with grad , batched with vmap , compiled with jit , and distributed with pmap or related sharding APIs. Functional Numerical...

Writes › Book › Auto Diff › Chapter 20. Building an AD Engine ›

Memory Management

Memory Management Memory management is the main systems problem in reverse mode automatic differentiation. The derivative rules are usually small. The hard part is deciding which primal values, intermediate tensors, gradients, graph nodes, and tape records must stay alive until the backward pass finishes. A minimal engine can ignore most of this at first. It can allocate every intermediate value and keep everything until Backward returns. That design is correct,...

Writes › Book › Auto Diff › Chapter 19. Theory and Foundations ›

Lambda Calculus and AD

Lambda Calculus and AD Automatic differentiation becomes substantially more difficult once programs contain higher-order functions. A first-order numerical program computes values: $$ f : \mathbb{R}^n \to \mathbb{R}^m. $$ A higher-order program computes functions, consumes functions, or returns functions: $$ F : (X \to Y) \to Z. $$ Ordinary AD systems built around tapes and computational graphs work well for first-order programs, but higher-order functions introduce semantic complications: closures, lexical environments,...

Writes › Book › Auto Diff › Chapter 18. Advanced Topics ›

Neural ODEs

Neural ODEs Classical neural networks apply a finite sequence of transformations: $$ h_{k+1} = f_k(h_k,\theta_k). $$ Depth corresponds to the number of layers. A neural ordinary differential equation (Neural ODE) replaces this discrete stack with continuous dynamics: $$ \frac{dz(t)}{dt} = f_\theta(z(t), t). $$ Instead of evaluating a finite composition, the model integrates a differential equation from an initial state to a final time. The output is $$ z(t_1), $$ obtained...

Writes › Book › Auto Diff › Chapter 17. Numerical and Systems Concerns ›

Gradient Vanishing and Explosion

Gradient Vanishing and Explosion Gradient-based optimization relies on propagating derivative information through many layers, time steps, or computational transformations. In deep systems, these gradients often become extremely small or extremely large during reverse propagation. These two pathologies are known as: vanishing gradients, exploding gradients. They are among the most important stability problems in automatic differentiation and deep learning. The phenomenon is fundamentally mathematical. Reverse mode propagates products of Jacobians backward...

Writes › Book › Auto Diff › Chapter 15. Differentiable Systems Architecture ›

Differentiable Search and Retrieval

Differentiable Search and Retrieval Differentiable search and retrieval systems integrate information access into gradient-based learning. Instead of treating retrieval as an external symbolic operation, the retrieval process becomes part of the computational graph. The system computes: $$ q \mapsto R(q) \mapsto y \mapsto L $$ where: Symbol Meaning $q$ Query representation $R(q)$ Retrieved information $y$ Downstream output $L$ Objective function Automatic differentiation propagates gradients backward through retrieval behavior, allowing the...

Writes › Book › Auto Diff › Chapter 14. Scientific Computing Applications ›

Molecular Simulation

Molecular Simulation Molecular simulation models the behavior of atoms and molecules using physical interaction laws. Automatic differentiation is important because many molecular methods require gradients of energy functions with respect to particle coordinates, force-field parameters, or quantum variables. A molecular system is typically described by particle positions $$ x = (x_1,\ldots,x_N), $$ where each particle position $$ x_i \in \mathbb{R}^3. $$ The system energy is $$ E(x,\theta), $$ where $\theta$...

Writes › Book › Auto Diff › Chapter 13. Optimization and Machine Learning ›

Sequence Models

Sequence Models Sequence models process ordered data. The input is not one independent vector, but a series: $$ x_1, x_2, \ldots, x_T. $$ Examples include text tokens, audio frames, video frames, time-series samples, event logs, trajectories, and program tokens. The model must represent both local content and position-dependent context. A sequence model usually computes hidden states: $$ h_t = f_\theta(x_t, h_{t-1}) $$ or contextual states: $$ h_t = f_\theta(x_1,\ldots,x_T,t). $$...

Writes › Book › Auto Diff › Chapter 12. AD in Modern Programming Languages ›

AD in Python

AD in Python Python became the dominant language for modern machine learning and differentiable computing because it combines a simple programming model with access to high-performance native libraries. Most Python automatic differentiation systems therefore follow a hybrid architecture: Layer Role Python frontend User-facing model and control logic Tensor runtime Dense array execution AD engine Gradient propagation Native backend CPU/GPU kernels in C/C++/CUDA Compiler subsystem Graph optimization and lowering Python itself...

Writes › Book › Auto Diff › Chapter 11. Compiler and Runtime Design ›

Graph IRs

Graph IRs A graph intermediate representation models a program as nodes and edges. Nodes represent operations or values. Edges represent data dependencies. For automatic differentiation, this form is natural because the chain rule also follows a dependency graph. A simple function def f(x): y = x * x z = sin(y) return z can be represented as: x -> mul -> y -> sin -> z The primal graph computes...

Writes › Book › Auto Diff › Chapter 10. Matrix and Tensor Differentiation ›

Differentiating Factorizations

Differentiating Factorizations Matrix factorizations rewrite a matrix into structured factors. They are used because the factors make later computations cheaper, more stable, or easier to interpret. In automatic differentiation, factorizations are delicate primitives because their outputs satisfy constraints. A generic matrix has free entries. A factorization output usually does not. Orthogonal matrices, triangular matrices, diagonal singular-value matrices, and permutation matrices all live in constrained spaces. The derivative must respect those...

Writes › Book › Auto Diff › Chapter 9. Differentiation of Control Flow ›

Piecewise Differentiability

Piecewise Differentiability A piecewise differentiable function is built from several differentiable pieces joined by boundaries. Each piece has an ordinary derivative inside its region. At the boundaries, the derivative may agree, disagree, or fail to exist. Conditionals, loops with data-dependent exits, clipping, thresholding, sorting, indexing, activation functions, and many numerical guards all produce piecewise differentiable programs. A typical example is: def f(x): if x < 0: return -x else: return...

Writes › Book › Auto Diff › Chapter 8. Higher-Order Differentiation ›

Nested AD

Nested AD Nested automatic differentiation means applying automatic differentiation inside another automatic differentiation computation. Conceptually, this means differentiating derivative computations. Examples include: Expression Meaning $\nabla(\nabla f)$ Hessian $D(Df)$ second directional derivative $\nabla(Df[v])$ Hessian-vector product $D(\nabla f)$ Jacobian of gradient $\nabla(\text{optimization step})$ meta-gradient Nested AD is essential for higher-order derivatives, implicit differentiation, meta-learning, differentiable optimization, and differentiable programming systems. The mathematics is straightforward. The implementation is subtle. First-Order AD as a...

Writes › Book › Auto Diff › Chapter 7. Dual Numbers and Algebraic Structures ›

Hyper-Dual Numbers

Hyper-Dual Numbers Dual numbers compute first derivatives exactly. Truncated polynomial algebras extend this to higher-order derivatives, but practical higher-order differentiation introduces an important problem: extracting second derivatives accurately without symbolic expansion or numerical cancellation. Hyper-dual numbers solve this problem by introducing multiple nilpotent infinitesimal directions whose mixed products survive. They provide an exact algebraic mechanism for computing: second derivatives mixed partial derivatives Hessians without finite differences and without truncation error....

Writes › Book › Auto Diff › Chapter 6. Reverse Mode Automatic Differentiation ›

Tape-Based Systems

Tape-Based Systems Most reverse mode automatic differentiation systems require a mechanism for recording the forward computation so that the reverse pass can later traverse it backward. This recorded structure is traditionally called a tape. The tape stores enough information to reconstruct local derivative relationships during reverse accumulation. Conceptually, the tape is an execution log of differentiable operations. Why Reverse Mode Needs a Tape Forward mode propagates tangents immediately during execution....

Writes › Book › Auto Diff › Chapter 5. Forward Mode Automatic Differentiation ›

Complexity Analysis

Complexity Analysis Forward mode automatic differentiation has a simple cost model. It evaluates the original program and, at the same time, evaluates the tangent program. Each primitive operation computes its primal result and its derivative contribution. For one tangent direction, the runtime is usually within a small constant factor of the original function evaluation. If the primal computation costs $$ C_f, $$ then one forward-mode derivative pass usually costs $$...

Writes › Book › Auto Diff › Chapter 4. Core Theory of Automatic Differentiation ›

Mixed-Mode Differentiation

Mixed-mode differentiation combines forward accumulation and reverse accumulation in the same derivative computation. It is used when neither pure forward mode nor pure reverse mode gives the best cost model. Forward mode computes Jacobian-vector products: $$ J_f(x)v $$ Reverse mode computes vector-Jacobian products: $$ u^T J_f(x) $$ Mixed mode composes these two operations to compute more structured derivatives efficiently. Why Mix Modes? For a function: $$ f:\mathbb{R}^n \to \mathbb{R}^m $$...

Writes › Book › Auto Diff › Chapter 3. Programs as Mathematical Objects ›

Loops and Recurrence Relations

Loops and Recurrence Relations Loops express repeated computation. Recurrence relations express the same idea mathematically: each state is computed from one or more earlier states. A loop such as h = h0 for t in range(n): h = tanh(w * h + b) y = h corresponds to the recurrence $$ h_{t+1} = \tanh(w h_t + b), \qquad y = h_n. $$ Automatic differentiation treats this as a sequence of...

Writes › Book › Auto Diff › Chapter 2. Mathematical Foundations ›

Computational Graphs

Computational Graphs A computational graph represents a calculation as nodes and edges. Nodes represent operations or values. Edges represent data dependencies. Automatic differentiation uses this graph structure to decide how derivative information should move through a computation. A simple program, a = x * y b = sin(a) c = b + x can be seen as the graph x ─┬─> multiply ─> a ─> sin ─> b ─┐ │...

Writes › Book › Auto Diff › Chapter 1. Introduction ›

Accuracy, Complexity, and Stability

Accuracy, Complexity, and Stability Derivative computation is not only a mathematical problem. It is also a numerical and systems problem. A derivative method must answer three questions simultaneously: Is the derivative accurate? How expensive is it? Is the computation numerically stable? These questions determine whether a differentiation method is practical for scientific computing, optimization, and machine learning. Automatic differentiation became important because it achieves a useful balance across all three...

Writes › Book › Auto Diff › Chapter 21. Major AD Systems ›

PyTorch Autograd

PyTorch Autograd PyTorch Autograd is a dynamic reverse-mode automatic differentiation system. It records tensor operations as they execute, builds a computation graph at runtime, and then traverses that graph backward to compute gradients. Its defining feature is dynamic graph construction. The graph follows the actual Python execution path taken by the program. This makes PyTorch convenient for models with conditionals, loops, recursion, variable-length inputs, and debugging workflows. Dynamic Computation Graphs...

Writes › Book › Auto Diff › Appendix ›

PyTorch Autograd

PyTorch Autograd PyTorch Autograd is a dynamic reverse-mode automatic differentiation system. It records tensor operations as they execute, builds a computation graph at runtime, and then traverses that graph backward to compute gradients. Its defining feature is dynamic graph construction. The graph follows the actual Python execution path taken by the program. This makes PyTorch convenient for models with conditionals, loops, recursion, variable-length inputs, and debugging workflows. Dynamic Computation Graphs...

Writes › Book › Auto Diff › Chapter 20. Building an AD Engine ›

Tape Design

Tape Design A tape is an append-only record of the operations executed during the forward pass. Reverse mode uses the tape to replay derivative rules backward. A graph stores nodes and edges as linked objects. A tape stores instructions in execution order. forward pass: input -> op 0 -> op 1 -> op 2 -> output tape: [op 0, op 1, op 2] backward pass: op 2 -> op 1...

Writes › Book › Auto Diff › Chapter 19. Theory and Foundations ›

Differential Categories

Differential Categories Cartesian differential categories model differentiation in categories with products. Differential categories generalize this idea further by shifting attention from cartesian structure to linear structure itself. The key observation is that differentiation is fundamentally linear. A derivative transforms nonlinear behavior into linear behavior: $$ f(x + \Delta x) \approx f(x) + Df(x)\Delta x. $$ This linear approximation is the essential object. Differential categories formalize: linear maps, derivative operators, resource-sensitive...

Writes › Book › Auto Diff › Chapter 18. Advanced Topics ›

Continuous-Time Adjoint Methods

Continuous-Time Adjoint Methods Many systems evolve continuously over time rather than through discrete layers. A state variable changes according to a differential equation: $$ \frac{dz(t)}{dt} = f(z(t), \theta, t). $$ The output is not a single value but an entire trajectory $$ z(t), \quad t \in [t_0, t_1]. $$ If a loss depends on the final state or on the full trajectory, we need derivatives with respect to parameters θ...

Writes › Book › Auto Diff › Chapter 17. Numerical and Systems Concerns ›

Memory Explosion

Memory Explosion Reverse-mode automatic differentiation trades computation for memory. To compute gradients efficiently, the backward pass requires access to intermediate values produced during the forward pass. For large computational graphs, storing these intermediates can dominate total system cost. This phenomenon is known as memory explosion. It is one of the central engineering constraints in modern automatic differentiation systems. Memory pressure limits: model size, batch size, sequence length, simulation resolution, graph...

Writes › Book › Auto Diff › Chapter 15. Differentiable Systems Architecture ›

Differentiable Physics Engines

Differentiable Physics Engines A differentiable physics engine computes gradients of physical simulation outputs with respect to inputs, parameters, or control signals. Instead of treating simulation as a terminal forward process, the simulator becomes part of the computational graph. The system computes: $$ x_{t+1} = f(x_t, u_t, \theta) $$ where: Symbol Meaning $x_t$ System state $u_t$ Control input $\theta$ Physical parameters $f$ Physics transition operator A loss over trajectories: $$ L(x_0,...

Writes › Book › Auto Diff › Chapter 14. Scientific Computing Applications ›

Computational Fluid Dynamics

Computational Fluid Dynamics Computational fluid dynamics studies fluid motion by solving discretized forms of the governing equations. Automatic differentiation enters CFD when we want gradients of simulation outputs with respect to geometry, boundary conditions, material parameters, control inputs, or model parameters. A CFD solver is usually a large numerical program: $$ \text{geometry} \to \text{mesh} \to \text{discrete equations} \to \text{linear and nonlinear solves} \to \text{flow field} \to \text{objective}. $$ The objective...

Writes › Book › Auto Diff › Chapter 13. Optimization and Machine Learning ›

Neural Network Training

Neural Network Training Neural network training is the repeated application of three operations: evaluate a model, differentiate a scalar loss, and update parameters. Automatic differentiation supplies the middle operation. The training system supplies data movement, batching, numerical policy, optimizer state, checkpointing, logging, and distributed execution. A minimal training step is: batch = next_batch(data) pred = model(batch.x) loss = loss_fn(pred, batch.y) grad = gradient(loss, model.parameters) optimizer.update(model.parameters, grad) This loop looks simple,...

Writes › Book › Auto Diff › Chapter 12. AD in Modern Programming Languages ›

AD in C and C++

AD in C and C++ C and C++ are important targets for automatic differentiation because much scientific, engineering, graphics, finance, and machine learning infrastructure is written in these languages. They provide direct control over memory layout, compilation, vectorization, and hardware-specific optimization. They also make AD difficult because programs can contain pointer aliasing, mutation, templates, manual memory management, macros, and calls into external libraries. In C and C++, automatic differentiation is...

Writes › Book › Auto Diff › Chapter 11. Compiler and Runtime Design ›

SSA Form

SSA Form Static single assignment form, or SSA, is an intermediate representation where each variable is assigned exactly once. A source program may reuse the same variable name many times: def f(x): x = x * x x = sin(x) return x In SSA, each assignment receives a fresh name: x0 = input x1 = x0 * x0 x2 = sin(x1) return x2 This small change has a large effect....

Writes › Book › Auto Diff › Chapter 10. Matrix and Tensor Differentiation ›

Linear Algebra Primitives

Linear Algebra Primitives Linear algebra primitives are tensor operations with algebraic structure: matrix multiplication, triangular solves, factorizations, inverses, determinants, norms, and spectral operations. They are central to automatic differentiation because many numerical programs are built from these operations rather than from scalar arithmetic alone. An AD system can differentiate these primitives in two ways. It can decompose them into smaller operations and apply generic AD, or it can attach a...

Writes › Book › Auto Diff › Chapter 9. Differentiation of Control Flow ›

Dynamic Graphs

Dynamic Graphs A dynamic graph is a computation graph built while the program runs. Its structure depends on ordinary runtime values: branches, loop counts, recursive calls, tensor shapes, data structures, and sometimes input data itself. In a static graph system, the graph is mostly known before execution. In a dynamic graph system, the graph is the execution. This matters for automatic differentiation because reverse mode needs to know which operations...

Writes › Book › Auto Diff › Chapter 8. Higher-Order Differentiation ›

Higher-Order Reverse Mode

Higher-Order Reverse Mode Reverse mode is efficient for scalar-output functions because it propagates one adjoint backward through the computation and produces a full gradient. For $$ f : \mathbb{R}^n \to \mathbb{R}, $$ one reverse pass computes $$ \nabla f(x) $$ at a cost comparable to a small constant multiple of evaluating $f$. Higher-order reverse mode asks for derivatives of reverse-mode derivative computations. The simplest case is differentiating the gradient: $$...

Writes › Book › Auto Diff › Chapter 7. Dual Numbers and Algebraic Structures ›

Truncated Polynomial Algebras

Truncated Polynomial Algebras Dual numbers capture first-order derivatives because the infinitesimal element satisfies $$ \varepsilon^2 = 0. $$ To compute higher-order derivatives, we extend this idea by allowing higher powers to survive temporarily before truncation. The resulting structures are called truncated polynomial algebras. They generalize dual numbers from first-order infinitesimals to finite-order infinitesimal expansions. From Dual Numbers to Higher Orders Recall the dual-number algebra: $$ \mathbb{R}[\varepsilon]/(\varepsilon^2). $$ Every element has...

Writes › Book › Auto Diff › Chapter 6. Reverse Mode Automatic Differentiation ›

Reverse Accumulation Algorithms

Reverse Accumulation Algorithms Reverse accumulation is the operational core of reverse mode automatic differentiation. The forward pass evaluates a program and records dependency information. The reverse pass traverses that recorded structure backward and accumulates adjoint contributions into each variable. The word accumulation is important. A variable may influence the final output through many downstream paths. Reverse mode computes the total derivative by summing contributions from all such paths. For a...

Writes › Book › Auto Diff › Chapter 5. Forward Mode Automatic Differentiation ›

Jacobian-Vector Products

Jacobian-Vector Products The natural output of forward mode automatic differentiation is a Jacobian-vector product. Instead of constructing the full Jacobian matrix explicitly, forward mode computes how a perturbation vector propagates through a function. For a function $$ f : \mathbb{R}^n \to \mathbb{R}^m, $$ the Jacobian at $x$ is $$ J_f(x) = \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \cdots & \frac{\partial f_1}{\partial x_n} \ \vdots & \ddots & \vdots \ \frac{\partial f_m}{\partial...

Writes › Book › Auto Diff › Chapter 4. Core Theory of Automatic Differentiation ›

Reverse Accumulation

Reverse accumulation is the reverse-mode form of automatic differentiation. It propagates derivative information backward from outputs to inputs. Forward mode asks: $$ \text{How does each intermediate value change when the input changes?} $$ Reverse mode asks: $$ \text{How does the final output change when each intermediate value changes?} $$ The second question is the one needed for gradients of scalar-output functions. The Adjoint Value Reverse mode attaches an adjoint to...

Writes › Book › Auto Diff › Chapter 3. Programs as Mathematical Objects ›

Control Flow

Control Flow Control flow determines which operations a program executes. Straight-line programs have a fixed sequence of operations, but ordinary programs contain branches, loops, recursion, early returns, and exceptions. Automatic differentiation must account for this structure without losing the basic rule: derivatives propagate through the operations that actually determine the output. A small example already shows the issue: if x > 0: y = x * x else: y =...

Writes › Book › Auto Diff › Chapter 2. Mathematical Foundations ›

Chain Rule as Composition Algebra

Chain Rule as Composition Algebra The chain rule is the central theorem behind automatic differentiation. Every useful AD algorithm is a disciplined way of applying the chain rule to a program. A program is built from smaller operations. Each operation maps input values to output values. When one operation feeds another, the corresponding mathematical functions are composed. AD differentiates the whole computation by differentiating each small mapping and composing their...

Writes › Book › Auto Diff › Chapter 1. Introduction ›

Automatic Differentiation

Automatic Differentiation Automatic differentiation computes derivatives by applying the chain rule to the operations of a program. The input is ordinary code that computes a value. The output is code, data, or runtime behavior that also computes derivative information. AD sits between numerical differentiation and symbolic differentiation. Numerical differentiation estimates derivatives from repeated function evaluations. Symbolic differentiation rewrites formulas into derivative formulas. Automatic differentiation follows the actual computation and propagates...

Writes › Book › Auto Diff › Appendix ›

TensorFlow Autograd

TensorFlow Autograd TensorFlow Autograd refers to TensorFlow’s automatic differentiation system, mainly exposed through tf.GradientTape . It is a reverse-mode AD system designed for tensor programs, neural network training, and differentiable numerical computation. TensorFlow began with static computational graphs. A user built a graph first, then executed it inside a runtime. Later TensorFlow 2 made eager execution the default. The modern interface records operations dynamically during execution, then differentiates the recorded...

Writes › Book › Auto Diff › Chapter 21. Major AD Systems ›

TensorFlow Autograd

TensorFlow Autograd TensorFlow Autograd refers to TensorFlow’s automatic differentiation system, mainly exposed through tf.GradientTape . It is a reverse-mode AD system designed for tensor programs, neural network training, and differentiable numerical computation. TensorFlow began with static computational graphs. A user built a graph first, then executed it inside a runtime. Later TensorFlow 2 made eager execution the default. The modern interface records operations dynamically during execution, then differentiates the recorded...

Writes › Book › Auto Diff › Chapter 20. Building an AD Engine ›

Graph Representation

Graph Representation A graph representation makes the structure of a differentiated computation explicit. In reverse mode, this structure is required because the backward pass must know which intermediate values depend on which earlier values. In compiler-based systems, the graph may also serve as an optimization target. A computation graph is a directed graph whose nodes represent values or operations, and whose edges represent data dependencies. For an expression such as:...

Writes › Book › Auto Diff › Chapter 19. Theory and Foundations ›

Categorical Semantics

Categorical Semantics Algebraic semantics describes differentiation through derivations, tangent maps, and linear structure. Categorical semantics goes further. It studies differentiation as a structural property of composition itself. Category theory is useful here because automatic differentiation is fundamentally compositional. Programs are built from smaller programs. Derivatives are built from smaller derivatives. The chain rule is a law about composition. The central question becomes: What categorical structure is required for differentiation to...

Writes › Book › Auto Diff › Chapter 18. Advanced Topics ›

Differentiable Optimization Layers

Differentiable Optimization Layers An optimization layer is a program component whose output is the solution of an optimization problem. Instead of computing $$ z = f(x) $$ by a fixed formula, it computes $$ z^*(\theta) = \arg\min_z L(z,\theta). $$ The parameters θ may come from a neural network, a simulator, a control system, or a statistical model. The output z* is then passed to another computation, often a scalar loss....

Writes › Book › Auto Diff › Chapter 17. Numerical and Systems Concerns ›

Overflow and Underflow

Overflow and Underflow Floating point systems represent numbers within a finite range. When a computed value exceeds the largest representable magnitude, overflow occurs. When a value becomes too small to represent accurately, underflow occurs. Automatic differentiation inherits both phenomena from the primal computation and may amplify them during derivative propagation. Overflow and underflow are not rare edge cases. They appear routinely in: deep neural networks, exponential models, probabilistic inference, scientific...

Writes › Book › Auto Diff › Chapter 15. Differentiable Systems Architecture ›

Differentiable Rendering

Differentiable Rendering Differentiable rendering is the process of computing derivatives of rendered images with respect to scene parameters. A renderer becomes part of the computational graph rather than a terminal visualization step. The central mapping is: $$ \theta \mapsto I $$ where: Symbol Meaning $\theta$ Scene parameters $I$ Rendered image The parameters may include: geometry camera pose material properties lighting texture maps volumetric density skeletal pose simulation state A loss...

Writes › Book › Auto Diff › Chapter 14. Scientific Computing Applications ›

Inverse Problems

Inverse Problems An inverse problem asks for causes from effects. A forward model predicts observations from parameters. An inverse model tries to recover parameters from observations. The usual form is $$ y = F(\theta), $$ where $\theta$ is an unknown parameter vector and $y$ is the model output. In practice, we observe noisy data $$ z \approx F(\theta). $$ The inverse problem is to estimate $\theta$ from $z$. Examples include...

Writes › Book › Auto Diff › Chapter 13. Optimization and Machine Learning ›

Backpropagation

Backpropagation Backpropagation is reverse mode automatic differentiation applied to neural networks. In most machine learning writing, the term refers to the whole training procedure: run a forward pass, compute a loss, propagate gradients backward, then update parameters. More precisely, backpropagation is the derivative computation step. A neural network is a composition of functions: $$ f_\theta(x) = f_k( f_{k-1}( \cdots f_2(f_1(x)) )). $$ Each layer produces an intermediate activation: $$ a_0...

Writes › Book › Auto Diff › Chapter 12. AD in Modern Programming Languages ›

Differentiable Programming

Differentiable Programming Differentiable programming treats differentiation as a general programming-language feature. A program can contain numerical kernels, control flow, data structures, solvers, simulations, and model code. If the program denotes a differentiable computation, the system should be able to compute derivatives of that program. This is broader than neural network training. A differentiable program may include a physics simulator, a renderer, a database query, an optimizer, or a probabilistic model....

Writes › Book › Auto Diff › Chapter 11. Compiler and Runtime Design ›

Intermediate Representations

Intermediate Representations An intermediate representation, or IR, is the internal program form used by a compiler or AD system after parsing and before final code generation. For automatic differentiation, the IR is where the system decides what the program means. It is also where differentiation becomes mechanical. A source program may contain syntax such as: def f(x): y = x * x z = sin(y) return z An AD compiler...

Writes › Book › Auto Diff › Chapter 10. Matrix and Tensor Differentiation ›

Broadcasting Semantics

Broadcasting Semantics Broadcasting is the rule system that allows tensor operations between arrays of different shapes without explicitly materializing expanded copies. It is one of the most important structural features in modern tensor systems. It is also one of the most common sources of gradient bugs. From a mathematical perspective, broadcasting defines an implicit linear replication operator. From a systems perspective, broadcasting defines a virtual tensor view with repeated values...

Writes › Book › Auto Diff › Chapter 9. Differentiation of Control Flow ›

Recursion

Recursion Recursion is control flow where a function calls itself. In automatic differentiation, recursion behaves like a loop with a call stack. Each recursive call contributes one layer to the computation, and differentiation applies the chain rule across the call tree. A simple recursive function has the form: def f(x, n): if n == 0: return x return g(f(x, n - 1)) This denotes repeated composition: $$ f(x,n) = g^{(n)}(x)...

Writes › Book › Auto Diff › Chapter 8. Higher-Order Differentiation ›

Hessian-Vector Products

Hessian-Vector Products A Hessian-vector product computes $$ H_f(x)v = \nabla^2 f(x)v, $$ where $$ f : \mathbb{R}^n \to \mathbb{R}, $$ and $$ v \in \mathbb{R}^n. $$ This operation applies the Hessian as a linear operator to a vector without explicitly constructing the full Hessian matrix. For large systems, this distinction is fundamental. A dense Hessian requires $O(n^2)$ storage. A Hessian-vector product requires only $O(n)$ storage for the input and output...

Writes › Book › Auto Diff › Chapter 7. Dual Numbers and Algebraic Structures ›

Geometric Interpretation

Geometric Interpretation Dual numbers provide an algebraic mechanism for differentiation, but they also have a precise geometric meaning. A dual number represents a point together with an infinitesimal displacement. Forward mode automatic differentiation can therefore be interpreted as the propagation of tangent vectors through a computation. This geometric viewpoint explains why the chain rule appears naturally, why derivatives are linear maps, and why forward mode computes directional derivatives. Points and...

Writes › Book › Auto Diff › Chapter 6. Reverse Mode Automatic Differentiation ›

Vector-Jacobian Products

Vector-Jacobian Products Reverse mode automatic differentiation fundamentally computes vector-Jacobian products. The gradient of a scalar function is a special case of this more general operation. Consider a differentiable function $$ f : \mathbb{R}^n \to \mathbb{R}^m. $$ Its Jacobian matrix is $$ J_f(x) = \frac{\partial f}{\partial x} = \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \cdots & \frac{\partial f_1}{\partial x_n} \ \vdots & \ddots & \vdots \ \frac{\partial f_m}{\partial x_1} & \cdots &...

Writes › Book › Auto Diff › Chapter 5. Forward Mode Automatic Differentiation ›

Forward Evaluation Rules

Forward Evaluation Rules Forward mode automatic differentiation works by replacing each primitive operation with an extended operation on pairs: $$ x \mapsto (x, \dot{x}) $$ The first component is the primal value. The second component is the tangent value. The evaluation rule for each operation must compute both. For a primitive operation $$ z = \phi(x_1, x_2, \ldots, x_k), $$ the forward rule is $$ \dot{z} = \sum_{i=1}^{k} \frac{\partial \phi}{\partial...

Writes › Book › Auto Diff › Chapter 4. Core Theory of Automatic Differentiation ›

Forward Accumulation

Forward accumulation is the forward-mode form of automatic differentiation. It propagates derivative information in the same order as ordinary program evaluation. Each intermediate value carries two pieces of information: Component Meaning Primal the ordinary value computed by the program Tangent the derivative of that value with respect to a chosen input direction If a program computes: $$ y=f(x) $$ then forward accumulation computes both: $$ y $$ and: $$ \dot...

Writes › Book › Auto Diff › Chapter 3. Programs as Mathematical Objects ›

Dependency Graphs

Dependency Graphs A dependency graph describes how values in a computation depend on earlier values. Automatic differentiation operates on these dependencies. For a straight-line program, v1 = x1 v2 = x2 v3 = v1 * v2 v4 = sin(v3) v5 = v4 + v1 y = v5 the dependency structure is: v1 ─┬─> v3 ─> v4 ─┐ │ ├─> v5 v2 ─┘ │ v1 ───────────────┘ Each node represents a variable...

Writes › Book › Auto Diff › Chapter 2. Mathematical Foundations ›

Jacobians and Hessians

Jacobians and Hessians The gradient is enough when a function has many inputs and one scalar output. More general programs need more general derivative objects. Two of the most important are the Jacobian and the Hessian. The Jacobian stores first-order derivatives of a vector-valued function. The Hessian stores second-order derivatives of a scalar-valued function. Both can be large. In automatic differentiation, we often avoid constructing them explicitly. Instead, we compute...

Writes › Book › Auto Diff › Chapter 1. Introduction ›

Symbolic Differentiation

Symbolic Differentiation Symbolic differentiation computes derivatives by manipulating expressions. The input is a formula. The output is another formula. For example, given $$ f(x) = x^2 + \sin x, $$ symbolic differentiation produces $$ f'(x) = 2x + \cos x. $$ This is the form of differentiation taught in calculus courses. It is exact over the algebraic expression being manipulated. It uses rules such as the sum rule, product rule,...

Writes › Book › Auto Diff › Appendix ›

Tapenade

Tapenade Tapenade is a source-transformation automatic differentiation system developed at INRIA. Like ADIFOR, it takes an existing program and produces a new differentiated program. Its main targets are Fortran and C, with strong emphasis on scientific and engineering codes. Tapenade is historically important because it pushed source-transformation AD beyond simple forward propagation. It supports both tangent mode and adjoint mode, making it useful for sensitivity analysis, inverse problems, and large...

Writes › Book › Auto Diff › Chapter 21. Major AD Systems ›

Tapenade

Tapenade Tapenade is a source-transformation automatic differentiation system developed at INRIA. Like ADIFOR, it takes an existing program and produces a new differentiated program. Its main targets are Fortran and C, with strong emphasis on scientific and engineering codes. Tapenade is historically important because it pushed source-transformation AD beyond simple forward propagation. It supports both tangent mode and adjoint mode, making it useful for sensitivity analysis, inverse problems, and large...

Writes › Book › Auto Diff › Chapter 20. Building an AD Engine ›

Minimal Reverse Mode Engine

Minimal Reverse Mode Engine Reverse mode automatic differentiation computes derivatives by traversing the program backward after evaluation. Unlike forward mode, which propagates tangents alongside values, reverse mode separates computation into two phases: Forward evaluation Backward gradient propagation The forward pass computes values and records dependency information. The backward pass propagates sensitivities from outputs toward inputs. For a scalar-valued function $$ f : \mathbb{R}^n \to \mathbb{R} $$ reverse mode computes the...

Writes › Book › Auto Diff › Chapter 19. Theory and Foundations ›

Algebraic Semantics

Algebraic Semantics Automatic differentiation is often introduced operationally. A program executes elementary operations, and derivative information propagates alongside the computation. This operational view explains implementation, but it does not explain what differentiation fundamentally means at the semantic level. Algebraic semantics studies differentiation as a structure-preserving transformation between programs, functions, and algebraic objects. Instead of viewing AD as a collection of local derivative rules, it treats differentiation as an algebraic operation...

Writes › Book › Auto Diff › Chapter 18. Advanced Topics ›

Differentiating Through Solvers

Differentiating Through Solvers A solver is a program that computes a value by search, iteration, or factorization. Instead of evaluating a closed-form expression, it finds a value that satisfies a condition. Common examples include: Solver type Output condition Linear solver Ax = b Nonlinear root solver F(x, θ) = 0 Fixed-point solver x = G(x, θ) Optimizer ∇x L(x, θ) = 0 ODE solver trajectory satisfies differential equation Projection operator...

Writes › Book › Auto Diff › Chapter 17. Numerical and Systems Concerns ›

Stability of Reverse Mode

Stability of Reverse Mode Reverse mode automatic differentiation computes gradients by propagating adjoint values backward through a computational graph. In exact arithmetic, the reverse accumulation rules are mathematically correct applications of the chain rule. In finite precision arithmetic, however, reverse propagation can become numerically unstable. The instability does not usually come from reverse mode itself. It emerges from the interaction between: floating point rounding, ill-conditioned computations, long dependency chains, repeated...

Writes › Book › Auto Diff › Chapter 15. Differentiable Systems Architecture ›

Differentiable Databases

Differentiable Databases A differentiable database is a data system whose operations participate in gradient-based optimization. Instead of treating storage and querying as external infrastructure, the database becomes part of the computational graph. The central idea is: $$ \text{query} \rightarrow \text{retrieval} \rightarrow \text{transformation} \rightarrow \text{loss} $$ with gradients propagating backward through retrieval, ranking, aggregation, filtering, and learned representations. This changes the role of the database. A traditional database answers queries exactly....

Writes › Book › Auto Diff › Chapter 14. Scientific Computing Applications ›

Sensitivity Analysis

Sensitivity Analysis Sensitivity analysis studies how changes in inputs affect the outputs of a system. In differential equations, optimization, simulation, and machine learning, the main object is usually a function $$ y = f(x,\theta), $$ where $x$ is an input or initial condition and $\theta$ is a parameter vector. The central question is: $$ \text{How does } y \text{ change when } x \text{ or } \theta \text{ changes?} $$...

Writes › Book › Auto Diff › Chapter 13. Optimization and Machine Learning ›

Stochastic Optimization

Stochastic Optimization Stochastic optimization studies optimization when the objective is accessed through samples, noisy estimates, or partial observations. In machine learning, this is the normal case. We do not usually optimize the loss over the entire data distribution. We optimize an empirical approximation, and even that empirical loss is usually accessed through mini-batches. The ideal objective is often written as an expectation: $$ L(\theta) = \mathbb{E} {(x,y) \sim P}[\ell(f \theta(x),...

Writes › Book › Auto Diff › Chapter 12. AD in Modern Programming Languages ›

Functional Languages

Functional Languages Functional programming languages provide a natural semantic foundation for automatic differentiation. Programs are expressed as compositions of functions, immutable values, and transformations over data. This aligns closely with the mathematical structure of differentiation. In a pure functional language, a function behaves like a mathematical mapping: $$ f : X \rightarrow Y $$ The absence of mutable state means evaluation can be analyzed as a composition graph. Automatic differentiation...

Writes › Book › Auto Diff › Chapter 11. Compiler and Runtime Design ›

Operator Overloading

Operator Overloading Operator overloading implements automatic differentiation by changing the meaning of ordinary arithmetic operations for special numeric objects. Instead of rewriting the program, the AD system gives numbers extra derivative state. When the user writes: y = sin(x * x) the operations * and sin execute derivative-aware methods. The program looks unchanged, but the values flowing through it are no longer plain scalars. They are AD values. A forward-mode...

Writes › Book › Auto Diff › Chapter 10. Matrix and Tensor Differentiation ›

Tensor Operations

Tensor Operations Tensor operations generalize scalar, vector, and matrix operations to arrays with arbitrary rank. In automatic differentiation, a tensor is usually treated as a typed array with shape, dtype, layout, and device. The mathematical object is an element of a finite-dimensional vector space. The systems object is a strided memory view with rules for indexing and mutation. A tensor $X$ with shape $$ (d_1, d_2, \ldots, d_k) $$ has...

Writes › Book › Auto Diff › Chapter 9. Differentiation of Control Flow ›

forward

Loops A loop repeats a computation until a condition fails or a fixed iteration count is reached. In automatic differentiation, loops are important because many numerical algorithms are fundamentally iterative: optimization methods recurrent neural networks numerical solvers simulations dynamic programming graph algorithms A loop introduces repeated dependence between program states. The derivative of the loop therefore depends on how perturbations propagate through the sequence of iterations. A loop can be...

Writes › Book › Auto Diff › Chapter 8. Higher-Order Differentiation ›

Hessian Computation

Hessian Computation For a scalar function $$ f : \mathbb{R}^n \to \mathbb{R}, $$ the Hessian is the matrix of all second partial derivatives: $$ H_f(x) = \nabla^2 f(x). $$ Its entries are $$ (H_f(x))_{ij} = \frac{\partial^2 f}{\partial x_i \partial x_j}(x). $$ Computing the Hessian means computing the derivative of the gradient: $$ \nabla^2 f(x) = D(\nabla f)(x). $$ This definition is simple. The difficulty is cost. A gradient has $n$...

Writes › Book › Auto Diff › Chapter 7. Dual Numbers and Algebraic Structures ›

Nilpotent Elements

Nilpotent Elements The defining feature of dual numbers is the existence of a nonzero element whose square vanishes: $$ \varepsilon^2 = 0. $$ Such elements are called nilpotent elements. Nilpotent structure is what allows automatic differentiation to isolate first-order behavior while discarding higher-order terms automatically. The algebra of dual numbers is therefore best understood as a special case of a more general idea: extending ordinary arithmetic with nilpotent infinitesimals. Definition...

Writes › Book › Auto Diff › Chapter 6. Reverse Mode Automatic Differentiation ›

Reverse Computational Graphs

Reverse Computational Graphs Reverse mode automatic differentiation operates on a computational graph. The forward pass evaluates the graph from inputs to outputs. The reverse pass traverses the same graph in the opposite direction and propagates adjoints backward through the dependencies. A computational graph converts a program into a directed acyclic structure of values and operations. Each node represents either: a variable or intermediate value; an elementary operation. Edges represent data...

Writes › Book › Auto Diff › Chapter 5. Forward Mode Automatic Differentiation ›

Dual Numbers

Dual Numbers Dual numbers give forward mode automatic differentiation a compact algebraic form. Instead of storing a value and a tangent as two unrelated fields, we package them into one object: $$ x + \epsilon \dot{x} $$ where $\epsilon$ is a formal symbol satisfying $$ \epsilon^2 = 0. $$ The number $x$ is the primal value. The number $\dot{x}$ is the tangent. The symbol $\epsilon$ marks the tangent part. A...

Writes › Book › Auto Diff › Chapter 4. Core Theory of Automatic Differentiation ›

Elementary Operations

Automatic differentiation reduces differentiation to a finite collection of elementary operations. Every program, regardless of complexity, is decomposed into primitive computational steps with known local derivative rules. An AD system therefore requires two components: a representation of computation as primitive operations derivative propagation rules for each primitive This section formalizes these elementary operations and shows how derivative rules are attached to them. Primitive Operations A primitive operation is an operation...

Writes › Book › Auto Diff › Chapter 3. Programs as Mathematical Objects ›

Intermediate Variables

Intermediate Variables Intermediate variables are the named values created between program inputs and program outputs. They make automatic differentiation mechanical. Consider: y = sin(x1 * x2) + exp(x2) A straight-line version is: v1 = x1 v2 = x2 v3 = v1 * v2 v4 = sin(v3) v5 = exp(v2) v6 = v4 + v5 y = v6 The expression has been decomposed into elementary assignments. Each assignment has one local...

Writes › Book › Auto Diff › Chapter 2. Mathematical Foundations ›

Multivariate Calculus

Multivariate Calculus Automatic differentiation is usually applied to functions with many inputs and many outputs. The calculus needed for this setting is multivariate calculus: the study of how a quantity changes when several variables change at once. A scalar function of several variables has the form $$ f : \mathbb{R}^n \to \mathbb{R} $$ For example, $$ f(x_1, x_2) = x_1^2 + 3x_1x_2 + \sin x_2 $$ depends on two independent...

Writes › Book › Auto Diff › Chapter 1. Introduction ›

Numerical Differentiation

Numerical Differentiation Numerical differentiation estimates derivatives by evaluating a function at nearby input values. It treats the function as a black box. The method does not need access to the internal structure of the program, only the ability to call it. For a scalar function $$ f : \mathbb{R} \to \mathbb{R}, $$ the simplest finite difference approximation is $$ f'(x) \approx \frac{f(x+h)-f(x)}{h}. $$ Here $h$ is a small perturbation. The...

Writes › Book › Auto Diff › Appendix ›

Appendix

ADIFOR ADIFOR, short for Automatic Differentiation of Fortran, is one of the classical source-transformation systems for automatic differentiation. It was designed for numerical programs written in Fortran, especially scientific computing codes where derivatives were needed for optimization, sensitivity analysis, and parameter estimation. The central idea of ADIFOR is simple: take an existing Fortran program that computes a function, analyze its source code, and generate a new Fortran program that computes...

Writes › Book › Auto Diff › Chapter 21. Major AD Systems ›

Chapter 21. Major AD Systems

ADIFOR ADIFOR, short for Automatic Differentiation of Fortran, is one of the classical source-transformation systems for automatic differentiation. It was designed for numerical programs written in Fortran, especially scientific computing codes where derivatives were needed for optimization, sensitivity analysis, and parameter estimation. The central idea of ADIFOR is simple: take an existing Fortran program that computes a function, analyze its source code, and generate a new Fortran program that computes...

Writes › Book › Auto Diff › Chapter 20. Building an AD Engine ›

Chapter 20. Building an AD Engine

Minimal Forward Mode Engine A minimal forward mode automatic differentiation engine has one job: evaluate a program while carrying both a value and its derivative. The engine does not build a graph. It does not store a tape. It computes derivatives in the same order as the original computation. Forward mode is best understood as ordinary evaluation over a richer number type. Instead of evaluating with real numbers, we evaluate...

Writes › Book › Auto Diff › Chapter 19. Theory and Foundations ›

Chapter 19. Theory and Foundations

Complexity Bounds Automatic differentiation is often described by a simple rule: The derivative of a program can usually be computed with cost comparable to the cost of running the program itself. This statement is useful, but incomplete. The cost depends on the shape of the function, the derivative object requested, the accumulation mode, memory behavior, and the representation of intermediate values. A scalar derivative, a gradient, a full Jacobian, and...

Writes › Book › Auto Diff › Chapter 18. Advanced Topics ›

Chapter 18. Advanced Topics

Implicit Differentiation Many programs do not compute their output by applying a fixed sequence of explicit operations. Instead, they define the output as the solution of another problem. A linear solver returns x such that $$ Ax=b. $$ An optimizer returns z such that $$ \nabla_z L(z,\theta)=0. $$ A fixed-point solver returns x such that $$ x = F(x,\theta). $$ In all three cases, the output depends on input parameters,...

Writes › Book › Auto Diff › Chapter 17. Numerical and Systems Concerns ›

Chapter 17. Numerical and Systems Concerns

Floating Point Error Automatic differentiation computes derivatives by executing arithmetic. On a real machine, arithmetic uses finite precision. This means AD gives the derivative of the implemented floating point program, not always the exact derivative of the ideal real-valued mathematical function. That distinction matters. Suppose a program represents a function $$ f : \mathbb{R}^n \to \mathbb{R}^m. $$ Mathematically, we may want the Jacobian $$ J_f(x) = \frac{\partial f(x)}{\partial x}. $$...

Writes › Book › Auto Diff › Chapter 15. Differentiable Systems Architecture ›

Chapter 15. Differentiable Systems Architecture

End-to-End Differentiable Pipelines An end-to-end differentiable pipeline is a system whose final objective can send derivative information backward through every trainable or tunable stage of computation. Instead of treating a model as the only differentiable component, the pipeline treats preprocessing, representation, simulation, retrieval, rendering, scoring, and decision rules as parts of one computational object. The basic form is: $$ x \mapsto z_1 \mapsto z_2 \mapsto \cdots \mapsto z_k \mapsto y...

Writes › Book › Auto Diff › Chapter 14. Scientific Computing Applications ›

Chapter 14. Scientific Computing Applications

Differential Equations Differential equations are one of the main reasons automatic differentiation matters in scientific computing. Many scientific models are not written as closed-form functions. They are written as equations describing how a system changes over time, space, or both. A simple ordinary differential equation has the form $$ \frac{dy(t)}{dt} = f(y(t), t, \theta), $$ where $y(t)$ is the state, $t$ is time, and $\theta$ is a vector of parameters....

Writes › Book › Auto Diff › Chapter 13. Optimization and Machine Learning ›

Chapter 13. Optimization and Machine Learning

Gradient Descent Gradient descent is the basic optimization procedure behind much of modern machine learning. It is simple enough to state in one line, but rich enough to expose many of the main issues in automatic differentiation: representation of objectives, computation of gradients, floating point behavior, memory use, batching, and convergence. Given a differentiable scalar objective $$ L(\theta): \mathbb{R}^n \to \mathbb{R}, $$ gradient descent updates the parameter vector $\theta$ by...

Writes › Book › Auto Diff › Chapter 12. AD in Modern Programming Languages ›

Chapter 12. AD in Modern Programming Languages

Lisp Systems Lisp is one of the natural homes of automatic differentiation. It treats programs as data, has a simple expression syntax, and supports macro systems that can transform code before execution. These properties make it possible to implement automatic differentiation as a direct program transformation rather than as an external compiler pass. A Lisp system represents code as symbolic expressions. The expression (* (+ x 1) (sin x)) is...

Writes › Book › Auto Diff › Chapter 11. Compiler and Runtime Design ›

Chapter 11. Compiler and Runtime Design

Source Transformation Source transformation is an implementation strategy for automatic differentiation in which a program that computes a function is rewritten into another program that computes derivatives of that function. The input is source code, or an intermediate representation close enough to source code. The output is new code with derivative computation built into it. For example, a function y = f(x) may be transformed into a function (y, dy)...

Writes › Book › Auto Diff › Chapter 10. Matrix and Tensor Differentiation ›

Chapter 10. Matrix and Tensor Differentiation

Matrix Calculus Matrix calculus is the notation and rule system used to differentiate functions whose inputs, outputs, or intermediate values are vectors, matrices, or tensors. Automatic differentiation systems do not require matrix calculus internally: they only need local derivative rules for primitive operations. However, matrix calculus is the language used to specify those rules, check them, and understand their shape. A scalar function has the familiar form $$ f :...

Writes › Book › Auto Diff › Chapter 9. Differentiation of Control Flow ›

Chapter 9. Differentiation of Control Flow

Conditionals A conditional is a program construct that chooses one computation among several possible computations. In ordinary code, this is written as if , else , switch , case , pattern matching, or guarded expressions. In automatic differentiation, a conditional raises a simple question: When a program chooses one branch, which derivative should the differentiated program compute? For a fixed execution, the answer is direct. The derivative follows the branch...

Writes › Book › Auto Diff › Chapter 8. Higher-Order Differentiation ›

Chapter 8. Higher-Order Differentiation

Second Derivatives First derivatives describe local rate of change. Second derivatives describe how that rate of change itself changes. In optimization, this is curvature. In dynamics, it is acceleration. In sensitivity analysis, it tells us whether a response is stable, amplifying, or changing direction. For a scalar function $$ f : \mathbb{R} \to \mathbb{R}, $$ the first derivative is $$ f'(x), $$ and the second derivative is $$ f''(x) =...

Writes › Book › Auto Diff › Chapter 7. Dual Numbers and Algebraic Structures ›

Chapter 7. Dual Numbers and Algebraic Structures

Algebra of Dual Numbers Dual numbers give the cleanest algebraic model of forward mode automatic differentiation. They extend ordinary real numbers with a formal infinitesimal part. Instead of carrying only a value, a dual number carries a value and its first-order variation. A dual number has the form $$ a + b\varepsilon $$ where $a,b \in \mathbb{R}$, and $\varepsilon$ is a formal element satisfying $$ \varepsilon^2 = 0 $$ but...

Writes › Book › Auto Diff › Chapter 6. Reverse Mode Automatic Differentiation ›

Chapter 6. Reverse Mode Automatic Differentiation

Adjoint Propagation Reverse mode automatic differentiation computes derivatives by propagating sensitivities backward through a computation. In forward mode, each intermediate value carries a tangent, which says how that value changes when the input is perturbed. In reverse mode, each intermediate value carries an adjoint, which says how the final output changes when that intermediate value is perturbed. Consider a scalar-valued function $$ f : \mathbb{R}^n \to \mathbb{R}. $$ A program...

Writes › Book › Auto Diff › Chapter 5. Forward Mode Automatic Differentiation ›

Chapter 5. Forward Mode Automatic Differentiation

Tangent Propagation Forward mode automatic differentiation computes derivatives by carrying two values through a program at the same time: the ordinary value and its tangent. The ordinary value tells us what the program computes. The tangent tells us how that value changes when the input is perturbed. For a scalar function $$ f : \mathbb{R} \to \mathbb{R}, $$ we usually write the derivative as $f'(x)$. Forward mode instead asks a...

Writes › Book › Auto Diff › Chapter 4. Core Theory of Automatic Differentiation ›

Chapter 4. Core Theory of Automatic Differentiation

Automatic differentiation is built on a simple observation: a complicated derivative can be computed by composing many small local derivatives. Instead of manipulating a full symbolic expression, or approximating derivatives numerically, AD propagates derivative information through each elementary operation in a program. This section develops the central mechanism underlying all forms of automatic differentiation: local derivative propagation. Programs as Compositions Consider a function: $$ f(x)=\sin(x^2+3x) $$ This function is not...

Writes › Book › Auto Diff › Chapter 3. Programs as Mathematical Objects ›

Chapter 3. Programs as Mathematical Objects

Straight-Line Programs A straight-line program is the simplest model of computation used in automatic differentiation. It is a program with a fixed sequence of assignments, no branches, no loops, and no recursion. Every operation is executed exactly once, in a known order. A typical straight-line program has the form: v1 = x1 v2 = x2 v3 = v1 * v2 v4 = sin(v3) v5 = v4 + v1 y =...

Writes › Book › Auto Diff › Chapter 2. Mathematical Foundations ›

Chapter 2. Mathematical Foundations

Functions and Mappings Automatic differentiation begins with a simple object: a function. A function maps an input value to an output value. In elementary calculus, this is often written as $$ f : \mathbb{R} \to \mathbb{R} $$ meaning that $f$ takes one real number and returns one real number. For example, $$ f(x) = x^2 + 3x + 1 $$ takes a scalar input $x$ and returns a scalar output....

Writes › Book › Auto Diff › Chapter 1. Introduction ›

Chapter 1. Introduction

Why Derivatives Matter A derivative measures how an output changes when an input changes. That sentence is simple, but it is one of the main ideas behind numerical computing, optimization, machine learning, simulation, control, statistics, and scientific modeling. When we write a function $$ y = f(x), $$ we usually care about more than the value of $y$. We also care about how sensitive $y$ is to $x$. If $x$...

Writes › Book › Auto Diff ›

Chapter 8. Higher-Order Differentiation

Section Title 1 Chapter 8. Higher-Order Differentiation 2 Hessian Computation 3 Hessian-Vector Products 4 Higher-Order Reverse Mode 5 Nested AD 6 Taylor Mode AD 7 Efficient Higher-Order Methods 8 Perturbation Confusion 9 Complexity of Higher Orders

Writes › Book › Auto Diff ›

Chapter 20. Building an AD Engine

Section Title 1 Chapter 20. Building an AD Engine 2 Minimal Reverse Mode Engine 3 Graph Representation 4 Tape Design 5 Memory Management 6 Operator Libraries 7 Custom Gradients 8 Performance Benchmarking 9 Testing Derivatives 10 Production Deployment

Writes › Book › Auto Diff ›

Chapter 14. Scientific Computing Applications

Section Title 1 Chapter 14. Scientific Computing Applications 2 Sensitivity Analysis 3 Inverse Problems 4 Computational Fluid Dynamics 5 Molecular Simulation 6 Computational Finance 7 Signal Processing 8 Robotics and Control 9 Probabilistic Programming

Writes › Book › Auto Diff ›

Chapter 17. Numerical and Systems Concerns

Section Title 1 Chapter 17. Numerical and Systems Concerns 2 Stability of Reverse Mode 3 Overflow and Underflow 4 Memory Explosion 5 Gradient Vanishing and Explosion 6 Determinism and Reproducibility 7 Parallelism 8 GPU and TPU Execution 9 Distributed Gradient Computation

Writes › Book › Auto Diff ›

Chapter 3. Programs as Mathematical Objects

Section Title 1 Chapter 3. Programs as Mathematical Objects 2 Intermediate Variables 3 Dependency Graphs 4 Control Flow 5 Loops and Recurrence Relations 6 Memory and State 7 Purity and Side Effects 8 Differentiable Subprograms

Writes › Book › Auto Diff ›

Chapter 10. Matrix and Tensor Differentiation

Section Title 1 Chapter 10. Matrix and Tensor Differentiation 2 Tensor Operations 3 Broadcasting Semantics 4 Linear Algebra Primitives 5 Differentiating Factorizations 6 Eigenvalue Problems 7 Singular Value Decomposition 8 Sparse Tensor Derivatives 9 GPU Tensor Kernels

Writes › Book › Auto Diff ›

Chapter 7. Dual Numbers and Algebraic Structures

Section Title 1 Chapter 7. Dual Numbers and Algebraic Structures 2 Nilpotent Elements 3 Geometric Interpretation 4 Truncated Polynomial Algebras 5 Hyper-Dual Numbers 6 Differential Algebras 7 Category-Theoretic View 8 Differential Lambda Calculus 9 Differential Lambda Calculus

Writes › Book › Auto Diff ›

Chapter 22. Open Problems

Section Title 1 Chapter 22. Open Problems

Writes › Book › Auto Diff ›

Chapter 6. Reverse Mode Automatic Differentiation

Section Title 1 Chapter 6. Reverse Mode Automatic Differentiation 2 Reverse Computational Graphs 3 Vector-Jacobian Products 4 Reverse Accumulation Algorithms 5 Tape-Based Systems 6 Wengert Lists 7 Memory-Time Tradeoffs 8 Checkpointing 9 Reverse Mode in Deep Learning 10 Case Studies

Writes › Book › Auto Diff ›

Chapter 5. Forward Mode Automatic Differentiation

Section Title 1 Chapter 5. Forward Mode Automatic Differentiation 2 Dual Numbers 3 Forward Evaluation Rules 4 Jacobian-Vector Products 5 Complexity Analysis 6 Higher-Dimensional Tangent Spaces 7 Efficient Seeding Strategies 8 Sparse Forward Methods 9 Case Studies

Writes › Book › Auto Diff ›

Chapter 16. Sparse and Structured Differentiation

Section Title 1 Chapter 16. Sparse and Structured Differentiation

Writes › Book › Auto Diff ›

Chapter 1. Introduction

Section Title 1 Chapter 1. Introduction 2 Numerical Differentiation 3 Symbolic Differentiation 4 Automatic Differentiation 5 Accuracy, Complexity, and Stability 6 Historical Development 7 Applications Across Science and Engineering

Writes › Book › Auto Diff ›

Chapter 4. Core Theory of Automatic Differentiation

Section Title 1 Chapter 4. Core Theory of Automatic Differentiation 2 Elementary Operations 3 Forward Accumulation 4 Reverse Accumulation 5 Mixed-Mode Differentiation 6 Computational Complexity 7 Numerical Exactness up to Floating Point 8 AD as Program Transformation 9 Summary and Synthesis 10 Tangent Propagation

Writes › Book › Auto Diff ›

Chapter 12. AD in Modern Programming Languages

Section Title 1 Chapter 12. AD in Modern Programming Languages 2 Functional Languages 3 Differentiable Programming 4 AD in C and C++ 5 AD in Python 6 AD in Rust 7 AD in Julia 8 AD in Swift 9 Type Systems for Differentiation 10 Effect Systems and Mutation

Writes › Book › Auto Diff ›

Chapter 18. Advanced Topics

Section Title 1 Chapter 18. Advanced Topics 2 Differentiating Through Solvers 3 Differentiable Optimization Layers 4 Continuous-Time Adjoint Methods 5 Neural ODEs 6 Probabilistic Automatic Differentiation 7 Quantum Differentiation 8 Differentiable Programming Languages 9 Verified Differentiation 10 Unified Differentiable Infrastructure

Writes › Book › Auto Diff ›

Appendix

Section Title 1 Appendix 2 Tapenade 3 TensorFlow Autograd 4 PyTorch Autograd 5 JAX 6 Zygote 7 Enzyme 8 Tinygrad 9 Comparative Architecture Analysis 10 Differentiation of Large Stateful Systems

Writes › Book › Auto Diff ›

Chapter 21. Major AD Systems

Section Title 1 Chapter 21. Major AD Systems 2 Tapenade 3 TensorFlow Autograd 4 PyTorch Autograd 5 JAX 6 Zygote 7 Enzyme 8 Tinygrad 9 Comparative Architecture Analysis 10 Differentiation of Large Stateful Systems

Writes › Book › Auto Diff ›

Chapter 11. Compiler and Runtime Design

Section Title 1 Chapter 11. Compiler and Runtime Design 2 Operator Overloading 3 Intermediate Representations 4 SSA Form 5 Graph IRs 6 Tracing Systems 7 Staging and Partial Evaluation 8 Memory Planning 9 Kernel Fusion 10 Ahead-of-Time vs Just-in-Time Differentiation

Writes › Book › Auto Diff ›

Chapter 13. Optimization and Machine Learning

Section Title 1 Chapter 13. Optimization and Machine Learning 2 Stochastic Optimization 3 Backpropagation 4 Neural Network Training 5 Sequence Models 6 Attention Mechanisms 7 Implicit Layers 8 Meta-Learning 9 Reinforcement Learning 10 Physics-Informed Models

Writes › Book › Auto Diff ›

Chapter 15. Differentiable Systems Architecture

Section Title 1 Chapter 15. Differentiable Systems Architecture 2 Differentiable Databases 3 Differentiable Rendering 4 Differentiable Physics Engines 5 Differentiable Search and Retrieval 6 Differentiable Compilers 7 Differentiable Operating Systems 8 Hybrid Symbolic-Numeric Systems 9 Summary

Writes › Book › Auto Diff ›

Chapter 9. Differentiation of Control Flow

Section Title 1 Chapter 9. Differentiation of Control Flow 2 forward 3 Recursion 4 Dynamic Graphs 5 Piecewise Differentiability 6 Non-Smooth Programs 7 Differentiating Stateful Systems 8 Exception Handling and Undefined Regions

Writes › Book › Auto Diff ›

Chapter 2. Mathematical Foundations

Section Title 1 Chapter 2. Mathematical Foundations 2 Multivariate Calculus 3 Jacobians and Hessians 4 Chain Rule as Composition Algebra 5 Computational Graphs 6 Linearization 7 Dual Spaces and Pushforwards 8 Taylor Expansions

Writes › Book ›

Auto Diff

Auto Diff Chapter Title Description 1 Chapter 1. Introduction A derivative measures how an output changes when an input changes. That sentence is simple, but it is one of the main ideas behind numerical computing, optimization, machine... 2 Chapter 2. Mathematical Foundations Automatic differentiation begins with a simple object: a function. 3 Chapter 3. Programs as Mathematical Objects A straight-line program is the simplest model of computation used in automatic...

Writes › Book › Auto Diff ›

Chapter 19. Theory and Foundations

Section Title 1 Chapter 19. Theory and Foundations 2 Algebraic Semantics 3 Categorical Semantics 4 Differential Categories 5 Lambda Calculus and AD 6 Program Equivalence 7 Formal Verification 8 Denotational Models 9 Differentiation as Functorial Transformation

#autodiff

Differentiation of Large Stateful Systems

Chapter 22. Open Problems

Differentiation of Large Stateful Systems

Production Deployment

Unified Differentiable Infrastructure

Chapter 16. Sparse and Structured Differentiation

Physics-Informed Models

Effect Systems and Mutation

Ahead-of-Time vs Just-in-Time Differentiation

Case Studies

Tangent Propagation

Taylor Expansions

Comparative Architecture Analysis

Comparative Architecture Analysis

Testing Derivatives

Differentiation as Functorial Transformation

Verified Differentiation

Distributed Gradient Computation

Summary

Probabilistic Programming

Reinforcement Learning

Type Systems for Differentiation

Kernel Fusion

GPU Tensor Kernels

Complexity of Higher Orders

Differential Lambda Calculus

Reverse Mode in Deep Learning

Case Studies

Summary and Synthesis

Tinygrad

Tinygrad

Performance Benchmarking

Denotational Models

Differentiable Programming Languages

GPU and TPU Execution

Hybrid Symbolic-Numeric Systems

Robotics and Control

Meta-Learning

AD in Swift

Memory Planning

Sparse Tensor Derivatives

Exception Handling and Undefined Regions

Perturbation Confusion

Differential Lambda Calculus

Checkpointing

Sparse Forward Methods

AD as Program Transformation

Differentiable Subprograms

Enzyme

Enzyme

Custom Gradients

Formal Verification

Quantum Differentiation

Parallelism

Differentiable Operating Systems

Signal Processing

Implicit Layers

AD in Julia

Staging and Partial Evaluation

Singular Value Decomposition

Differentiating Stateful Systems

Efficient Higher-Order Methods

Category-Theoretic View

Memory-Time Tradeoffs

Efficient Seeding Strategies

Numerical Exactness up to Floating Point

Purity and Side Effects

Dual Spaces and Pushforwards

Applications Across Science and Engineering

Zygote

Zygote

Operator Libraries

Program Equivalence

Probabilistic Automatic Differentiation

Determinism and Reproducibility

Differentiable Compilers

Computational Finance

Attention Mechanisms

AD in Rust