Chapter 21

Chapter 21 sections from Deep Learning with PyTorch.

7 items

Sections

  1. 21.1 Transformer Encoders
  2. 21.2 Transformer Decoders
  3. 21.3 Positional Encoding
  4. 21.4 Residual and Normalization Layers
  5. 21.5 Scaling Transformers
  6. 21.6 Efficient Transformers
  7. 21.7 Sparse Expert Architectures
Writes › Book › Deep Learning with PyTorch › Part VI › Chapter 21 ›

Transformer Encoders

A transformer encoder is a neural network block that maps a sequence of input vectors to a sequence of contextualized output vectors.
deep-learning pytorch
Writes › Book › Deep Learning with PyTorch › Part VI › Chapter 21 ›

Transformer Decoders

A transformer decoder is a neural network block that maps a prefix sequence to a sequence of next-token representations. It is used when the model must generate output one step at a time.
deep-learning pytorch
Writes › Book › Deep Learning with PyTorch › Part VI › Chapter 21 ›

Positional Encoding

Self-attention compares tokens to other tokens, but by itself it has no built-in notion of order.
deep-learning pytorch
Writes › Book › Deep Learning with PyTorch › Part VI › Chapter 21 ›

Residual and Normalization Layers

Transformer layers are deep stacks of attention and feedforward blocks.
deep-learning pytorch
Writes › Book › Deep Learning with PyTorch › Part VI › Chapter 21 ›

Scaling Transformers

Scaling a transformer means increasing its capacity, data exposure, context length, training compute, or serving throughput.
deep-learning pytorch
Writes › Book › Deep Learning with PyTorch › Part VI › Chapter 21 ›

Efficient Transformers

Standard transformer attention scales quadratically with sequence length. For a sequence of length $T$, self-attention constructs a score matrix of size
deep-learning pytorch
Writes › Book › Deep Learning with PyTorch › Part VI › Chapter 21 ›

Sparse Expert Architectures

Dense transformers activate every parameter for every token.
deep-learning pytorch