LLMs from Scratch
For ownersBuilding a language model end to end. It shows why a model knows nothing of your business until you hand it over.
Build, train, and understand large language models.
Start with lesson 1
Lesson 1 of 24, ~90 minutesTokenizers: BPE, WordPiece, SentencePieceThe lessons
- 01Tokenizers: BPE, WordPiece, SentencePieceA tokenizer splits text into pieces called tokens and turns them into numbers the model can read. Methods like BPE, WordPiece and SentencePiece decide where those splits fall.Build · ~90 minutes
- 02Building a Tokenizer from ScratchThis lesson builds a working BPE tokenizer by hand: training merges on text, handling special tokens and bytes, and encoding and decoding without losing information.Build · ~90 minutes
- 03Data Pipelines for Pre-TrainingA data pipeline collects, cleans, filters, removes duplicates from and packs the huge text collections used to pretrain a language model. Data quality shapes model quality more than most settings.Build · ~90 minutes
- 04Pre-Training a Mini GPT (124M Parameters)This lesson trains a small GPT-2 sized model from scratch on one GPU, showing the whole loop of predicting the next token, measuring loss and improving over many steps.Build · ~120 minutes
- 05Scaling: Distributed Training, FSDP, DeepSpeedWhen a model is too big for one GPU, training is split across many: copying data across machines, sharding weights and optimizer state, and splitting layers. FSDP and DeepSpeed handle much of this.Build · ~120 minutes
- 06Instruction Tuning (SFT)Supervised fine-tuning trains a base model on examples of instructions paired with good answers, turning a raw text predictor into an assistant that follows requests.Build · ~90 minutes
- 07RLHF: Reward Model + PPORLHF trains a reward model on human rankings of responses, then uses PPO to push the language model toward higher-scoring answers while staying close to its original behavior.Build · ~90 minutes
- 08DPO: Direct Preference OptimizationDirect Preference Optimization tunes a model on pairs of preferred and rejected answers with one simple loss, skipping the separate reward model and reinforcement learning step of RLHF.Build · ~90 minutes
- 09Constitutional AI and Self-ImprovementConstitutional AI gives a model a written list of principles, has it critique and revise its own answers against them, and trains on the results, cutting the need for human labels.Build · ~45 minutes
- 10Evaluation: Benchmarks, Evals, LM HarnessEvaluation means measuring what a model can actually do, using public benchmarks, custom test sets and tools like the LM Evaluation Harness, while watching for scores that were gamed.Build · ~90 minutes
- 11Quantization: Making Models FitQuantization stores model weights with fewer bits, such as 8 or 4 instead of 16, so models need less memory and run faster with a small drop in quality.Build · ~120 minutes
- 12Inference OptimizationInference optimization speeds up a running model. Reading the prompt can be done in parallel, while writing the answer goes one token at a time, and tricks like KV caching and batching target each part.Build · ~120 minutes
- 13Building a Complete LLM PipelineThis lesson ties every stage into one run: tokenizing, pretraining, scaling, fine-tuning, aligning, evaluating, compressing and serving a model, with checks between stages.Build · ~120 minutes
- 14Open Models: Architecture WalkthroughsA tour of how today's open models like Llama, Mistral, Qwen and DeepSeek differ from GPT-2, through a handful of design changes in normalization, activations, positions and attention.Learn · ~45 minutes
- 15Speculative Decoding and EAGLE-3EAGLE-3 is a modern speculative decoding method that trains a small draft head on the big model's internal features to guess several tokens ahead, which the big model then checks in one pass.Build · ~75 minutes
- 16Differential Attention (V2)Differential attention computes two attention maps and subtracts one from the other, cancelling the background noise that normal attention spreads across irrelevant tokens.Build · ~60 minutes
- 17Native Sparse Attention (DeepSeek NSA)Native Sparse Attention from DeepSeek lets a model look at a compressed summary, a few chosen blocks and a nearby window instead of every past token, making long inputs much cheaper.Build · ~60 minutes
- 18Multi-Token Prediction (MTP)Multi-token prediction trains a model to predict not just the next token but one or more after it, which gives a richer training signal and can speed up generation.Build · ~60 minutes
- 19DualPipe ParallelismDualPipe is DeepSeek's scheduling method for training across many GPUs that overlaps sending data between machines with computing, so GPUs spend less time waiting.Learn · ~60 minutes
- 20DeepSeek-V3 Architecture WalkthroughA walk through DeepSeek-V3's design: a large mixture-of-experts model where only a small part runs per token, plus compressed attention memory, multi-token prediction and low-precision training.Learn · ~75 minutes
- 21Jamba — Hybrid SSM-TransformerJamba mixes transformer layers with state space model layers, which handle long sequences in linear time and fixed memory, plus mixture of experts, to get long context at lower cost.Learn · ~60 minutes
- 22Async and Hogwild! InferenceHogwild! Inference runs several copies of the same model on one problem at once, all reading and writing one shared memory, so they can split the work without a fixed plan.Build · ~60 minutes
- 23Speculative Decoding and EAGLESpeculative decoding uses a small, fast model to guess several next tokens, then has the large model check them all in one pass, keeping only the ones it agrees with.Build · ~75 minutes
- 24Gradient Checkpointing and Activation RecomputationGradient checkpointing saves memory during training by keeping only some intermediate results and recomputing the rest when needed, trading extra compute for a smaller memory footprint.Build · ~70 minutes
Lessons from AI Engineering from Scratch by Rohit Ghumare and contributors, MIT license. Notes for owners by Lozo. License
Next step
Rather have it built for you?
Reading is optional. Tell me what's stuck and I'll build it into the business you already run.