Transformers Deep Dive

For ownersThe design inside today's chat models. The owner notes say what that means for what a model can do.

The architecture that changed everything. Understand every layer.

16 lessons

Start with lesson 1

Lesson 1 of 16, ~45 minutesWhy Transformers — The Problems with RNNs

The lessons

  1. 01Why Transformers — The Problems with RNNsOlder text models read one word at a time, which was slow and forgetful. Transformers look at every word at once, which let them train on far more data and grow much larger.Learn · ~45 minutes
  2. 02Self-Attention from ScratchSelf-attention lets each word in a sentence check every other word and decide how much each one matters to its own meaning. The lesson builds this step by step in code.Build · ~90 minutes
  3. 03Multi-Head AttentionMulti-head attention runs several attention steps side by side, each with its own settings. Each head can track a different kind of link between words, and the results are combined.Build · ~75 minutes
  4. 04Positional Encoding — Sinusoidal, RoPE, ALiBiAttention on its own ignores word order. Positional encoding adds information about where each word sits, using methods like sine waves, RoPE rotations or ALiBi distance penalties.Build · ~45 minutes
  5. 05The Full Transformer — Encoder + DecoderA full transformer stacks attention with supporting parts: feed-forward layers, shortcut connections, normalization, and in the original design, an encoder and a decoder linked by cross-attention.Build · ~75 minutes
  6. 06BERT — Masked Language ModelingBERT learns language by hiding random words in a sentence and guessing them from the words on both sides. It is built to understand text rather than write it.Build · ~45 minutes
  7. 07GPT — Causal Language ModelingGPT models learn by predicting the next word using only the words before it. A mask stops the model from peeking ahead. Repeating that prediction is how they write text.Build · ~75 minutes
  8. 08T5, BART — Encoder-Decoder ModelsT5 and BART pair an encoder that reads the full input with a decoder that writes the output. They are built for turning one text into another, like summaries or translations.Learn · ~45 minutes
  9. 09Vision Transformers (ViT)Vision transformers cut an image into small square patches and treat each patch like a word. The same attention method used for text then learns to recognize what is in the picture.Build · ~45 minutes
  10. 10Audio Transformers — Whisper ArchitectureWhisper turns audio into an image-like chart of sound frequencies over time, reads it with a transformer encoder, and writes out the words with a decoder.Learn · ~45 minutes
  11. 11Mixture of Experts (MoE)A mixture of experts model has many sub-networks, called experts, but only uses a few of them for each token. A router picks which experts handle each piece of input.Build · ~45 minutes
  12. 12KV Cache, Flash Attention & Inference OptimizationWhen a model writes text, it can save past work in a KV cache instead of redoing it for every new token. Flash attention computes attention in a way that moves less data in GPU memory.Build · ~75 minutes
  13. 13Scaling LawsScaling laws describe how a model's error drops in a predictable way as you add more parameters, more training data and more compute, and how to split a budget between them.Learn · ~45 minutes
  14. 14Build a Transformer from Scratch — The CapstoneThe capstone puts every earlier piece together into one working transformer written from scratch, then trains it and uses it to generate text.Build · ~120 minutes
  15. 15Attention Variants — Sliding Window, Sparse, DifferentialStandard attention compares every token with every other token, which gets costly on long inputs. Variants like sliding window, sparse and differential attention limit or sharpen what each token looks at.Build · ~60 minutes
  16. 16Speculative Decoding — Draft, Verify, RepeatSpeculative decoding speeds up text generation. A small fast model guesses several next tokens, and the big model checks them all in one pass, keeping the ones it agrees with.Build · ~60 minutes

Lessons from AI Engineering from Scratch by Rohit Ghumare and contributors, MIT license. Notes for owners by Lozo. License

Next step

Rather have it built for you?

Reading is optional. Tell me what's stuck and I'll build it into the business you already run.

Tell me what's stuck
  • A real conversation, not a pitch
  • You keep your accounts
  • Nothing sends without your yes