Chapter 13. Optimization and Machine Learning

Gradient descent is the basic optimization procedure behind much of modern machine learning. It is simple enough to state in one line, but rich enough to expose many of the...

10 items

Section	Title
1	Chapter 13. Optimization and Machine Learning
2	Stochastic Optimization
3	Backpropagation
4	Neural Network Training
5	Sequence Models
6	Attention Mechanisms
7	Implicit Layers
8	Meta-Learning
9	Reinforcement Learning
10	Physics-Informed Models

Writes › Book › Auto Diff › Chapter 13. Optimization and Machine Learning ›

Chapter 13. Optimization and Machine Learning

Gradient descent is the basic optimization procedure behind much of modern machine learning. It is simple enough to state in one line, but rich enough to expose many of the...

Writes › Book › Auto Diff › Chapter 13. Optimization and Machine Learning ›

Stochastic Optimization

Stochastic optimization studies optimization when the objective is accessed through samples, noisy estimates, or partial observations. In machine learning, this is the normal...

Writes › Book › Auto Diff › Chapter 13. Optimization and Machine Learning ›

Backpropagation

Backpropagation is reverse mode automatic differentiation applied to neural networks. In most machine learning writing, the term refers to the whole training procedure: run a...

Writes › Book › Auto Diff › Chapter 13. Optimization and Machine Learning ›

Neural Network Training

Neural network training is the repeated application of three operations: evaluate a model, differentiate a scalar loss, and update parameters. Automatic differentiation...

Writes › Book › Auto Diff › Chapter 13. Optimization and Machine Learning ›

Sequence Models

Sequence models process ordered data. The input is not one independent vector, but a series:

Writes › Book › Auto Diff › Chapter 13. Optimization and Machine Learning ›

Attention Mechanisms

Attention is a sequence operation that lets each position read information from other positions. Instead of compressing the whole past into one recurrent hidden state,...

Writes › Book › Auto Diff › Chapter 13. Optimization and Machine Learning ›

Implicit Layers

An implicit layer defines its output as the solution of an equation, not as a fixed sequence of explicit operations. Instead of computing

Writes › Book › Auto Diff › Chapter 13. Optimization and Machine Learning ›

Meta-Learning

Meta-learning studies systems that improve how they learn. Instead of only optimizing model parameters for one task, a meta-learning method optimizes some part of the learning...

Writes › Book › Auto Diff › Chapter 13. Optimization and Machine Learning ›

Reinforcement Learning

Reinforcement learning studies learning systems that act in an environment. Unlike supervised learning, the training signal is not a target label for each input. The model...

Writes › Book › Auto Diff › Chapter 13. Optimization and Machine Learning ›

Physics-Informed Models

Physics-informed models combine data fitting with equations from physics or applied mathematics. The model is trained not only to match observed samples, but also to satisfy...