Chapter 11 sections from Deep Learning with PyTorch.
8 items
A neural network is trained by minimizing a loss function. For a supervised learning problem, this loss measures how far the model predictions are from the target values.
Neural networks are usually trained iteratively. An optimizer repeatedly updates model parameters to reduce the training loss.
Dropout is a regularization method that randomly removes parts of a neural network during training.
Data augmentation is a regularization method that creates modified versions of training examples while preserving their labels.
Label smoothing is a regularization method for classification.
Stochastic depth is a regularization method for deep residual networks.
Mixup and CutMix are data augmentation methods that create new training examples by combining two examples and their labels. They regularize the model by discouraging overly sharp decision boundaries.
Stochastic depth regularizes deep residual networks by randomly skipping residual branches during training.