Chapter 15

Chapter 15 sections from Deep Learning with PyTorch.

7 items

Sections

  1. 15.1 Attention Mechanisms
  2. 15.2 Self-Attention
  3. 15.3 Multi-Head Attention
  4. 15.4 Positional Encoding
  5. 15.5 Transformer Encoders
  6. 15.6 Transformer Decoders
  7. 15.7 Efficient Attention Methods
Writes › Book › Deep Learning with PyTorch › Part IV › Chapter 15 ›

Attention Mechanisms

Attention is a method for letting a model choose which parts of an input are most relevant when producing an output.
deep-learning pytorch
Writes › Book › Deep Learning with PyTorch › Part IV › Chapter 15 ›

Self-Attention

Self-attention is attention applied within a single sequence. The same input supplies the queries, keys, and values. Each position builds a new representation by reading from other positions in the same sequence.
deep-learning pytorch
Writes › Book › Deep Learning with PyTorch › Part IV › Chapter 15 ›

Multi-Head Attention

Multi-head attention runs several attention operations in parallel.
deep-learning pytorch
Writes › Book › Deep Learning with PyTorch › Part IV › Chapter 15 ›

Positional Encoding

Self-attention compares tokens by content. By itself, it has no built-in notion of token order.
deep-learning pytorch
Writes › Book › Deep Learning with PyTorch › Part IV › Chapter 15 ›

Transformer Encoders

A transformer encoder is a stack of layers that maps an input sequence to a contextual sequence representation.
deep-learning pytorch
Writes › Book › Deep Learning with PyTorch › Part IV › Chapter 15 ›

Transformer Decoders

A transformer decoder maps a partial output sequence to predictions for the next token or next output step.
deep-learning pytorch
Writes › Book › Deep Learning with PyTorch › Part IV › Chapter 15 ›

Efficient Attention Methods

Standard self-attention compares every token with every other token. For a sequence of length $T$, this produces a $T \times T$ attention matrix. The cost grows quadratically with sequence length.
deep-learning pytorch