The path to large language models
Perceptron to GPT-3, in the order each one made the next one possible.
9 papers, in the order to read them. A paper that needs another one first is listed under it.
- 1The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain
a learning rule for a linear threshold unit
- 2Learning Representations by Back-propagating Errors
the chain rule applied to a layered network
Read first: The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain
- 3Gradient-Based Learning Applied to Document Recognition
convolution, weight sharing and pooling, trained end to end
Read first: Learning Representations by Back-propagating Errors
- 4ImageNet Classification with Deep Convolutional Neural Networks
depth, ReLU, dropout and two GPUs
Read first: Gradient-Based Learning Applied to Document Recognition
- 5Sequence to Sequence Learning with Neural Networks
an encoder LSTM into a decoder LSTM, with no alignment model
- 6Deep Residual Learning for Image Recognition
learn the residual, and a hundred layers become trainable
Read first: ImageNet Classification with Deep Convolutional Neural Networks
- 7Attention Is All You Need
self-attention alone, with no recurrence and no convolution
Read first: Sequence to Sequence Learning with Neural Networks
- 8BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
masked language modelling, then fine-tuning
Read first: Attention Is All You Need
- 9Language Models are Few-Shot Learners
scale, and a task specified in the prompt rather than the weights
Read first: Attention Is All You Need