Capstone Projects

For ownersFull projects that pull the course together. A good test for an engineer you plan to hire.

Prove everything you learned. Build portfolio-grade systems.

85 lessons

Start with lesson 1

Lesson 1 of 85, 35 hoursCapstone 01 — Terminal-Native Coding Agent

The lessons

  1. 01Capstone 01 — Terminal-Native Coding AgentA full project: build a coding assistant that runs in the terminal, plans a change, edits files, runs tests in a sandbox, recovers from errors and ends with a pull request.Capstone · 35 hours
  2. 02Capstone 02 — RAG over Codebase (Cross-Repo Semantic Search)A full project: build a search tool that understands what code means across many repositories, finds the right functions, and answers questions with links to the exact lines.Capstone · 30 hours
  3. 03Capstone 03 — Real-Time Voice Assistant (ASR to LLM to TTS)A full project: build a voice assistant that listens, understands, thinks and talks back fast enough to feel like a phone call, and lets the caller interrupt it.Capstone · 30 hours
  4. 04Capstone 04 — Multimodal Document QA (Vision-First PDF, Tables, Charts)A full project: build a system that answers questions about PDFs by looking at each page as an image, so tables, charts and scanned pages are read the way a person sees them.Capstone · 30 hours
  5. 05Capstone 05 — Autonomous Research Agent (AI-Scientist Class)A full project: build an agent that proposes research ideas, runs experiments in a sandbox, checks the results, writes a paper and has it reviewed by AI critics, all within a fixed budget.Capstone · 40 hours
  6. 06Capstone 06 — DevOps Troubleshooting Agent for KubernetesA full project: build an agent that wakes up on an alert, reads logs and metrics from a Kubernetes cluster, ranks likely causes and sends the team a short brief, with any fix waiting for human approval.Capstone · 30 hours
  7. 07Capstone 07 — End-to-End Fine-Tuning Pipeline (Data to SFT to DPO to Serve)A full project: take a mid-size open model, train it on your own examples, tune it toward preferred answers, shrink it for cheaper running and serve it, with a report of cost per token.Capstone · 35 hours
  8. 08Capstone 08 — Production RAG Chatbot for a Regulated VerticalA full project: build a question-answering chatbot over company documents for a regulated field, with hybrid search, reranking, guardrails, monitoring and graded test questions.Capstone · 30 hours
  9. 09Capstone 09 — Code Migration Agent (Repo-Level Language / Runtime Upgrade)A full project: build an agent that upgrades a whole codebase to a newer language or runtime version, using safe automatic rewrites first and AI only for the tricky parts, with tests on every branch.Capstone · 30 hours
  10. 10Capstone 10 — Multi-Agent Software Engineering TeamA full project: build a team of AI agents where one plans, several write code in parallel, one reviews and one tests, all sharing state and handing work along.Capstone · 40 hours
  11. 11Capstone 11 — LLM Observability & Eval DashboardA full project: build a dashboard that records every AI call, its cost, speed and quality scores, and lets a team search traces and run automatic evaluations on samples.Capstone · 25 hours
  12. 12Capstone 12 — Video Understanding Pipeline (Scene, QA, Search)A full project: build a pipeline that splits videos into scenes, describes and indexes each one with the transcript, and answers questions by pointing to the exact start and end times.Capstone · 30 hours
  13. 13Capstone 13: Stateless MCP Server with Registry and GovernanceA full project: build an MCP server that offers tools to AI clients, publish it to a registry, and add sign-in, permission rules and an audit trail for every call.Capstone · ~25 hours
  14. 14Capstone 14 — Speculative-Decoding Inference ServerA full project: build a model server where a small fast model guesses the next few words and the large model checks them in one pass, speeding up responses without changing the output.Capstone · 30 hours
  15. 15Capstone 15 — Constitutional Safety Harness + Red-Team RangeA full project: build a layered guardrail system around a model, with input and output classifiers, written rules, and a test range that attacks the system to find weak spots.Capstone · 25 hours
  16. 16Capstone 16 — GitHub Issue-to-PR Autonomous AgentA full project: build an agent that picks up a labeled GitHub issue, works on it in a cloud sandbox, checks that tests pass and opens a review-ready pull request with its reasoning.Capstone · 30 hours
  17. 17Capstone 17 — Personal AI Tutor (Adaptive, Multimodal, with Memory)A full project: build an AI tutor that guides learners with questions instead of handing out answers, tracks what each person knows, accepts voice, text and photos, and schedules reviews.Capstone · 30 hours
  18. 18Agent Harness Loop ContractThe first lesson of the agent harness track: define the loop that every agent runs, so any model can plug in while the harness controls steps, tools and stopping rules.Build · ~90 minutes
  19. 19Tool Registry with Schema ValidationBuild the list of tools an agent may use, each with a schema, and check every tool call against that schema before it runs.Build · ~90 minutes
  20. 20JSON-RPC 2.0 Over Newline-Delimited StdioBuild the message channel between an AI client and a tool server by hand: JSON-RPC 2.0 messages, one per line, sent over standard input and output.Build · ~90 minutes
  21. 21Function Call DispatcherBuild the part of an agent that actually runs a tool call, with timeouts, retries, duplicate detection and errors translated into messages the model can act on.Build · ~90 minutes
  22. 22Plan-Execute Control FlowBuild an agent that writes a plan, carries out each step, and rewrites the remaining plan when a step fails instead of giving up or blindly continuing.Build · ~90 minutes
  23. 23Capstone Lesson 25: Verification Gates and the Observation BudgetBuild a chain of checks that decides whether each tool call may run, limits how much output the agent gets to read, and stops the loop when it has read too much.Build · ~90 minutes
  24. 24Capstone Lesson 26: Sandbox Runner with Denylist and Path JailBuild a sandbox that runs agent commands with a list of blocked programs, keeps every file path inside the project folder, trims long output and kills anything that runs too long.Build · ~90 minutes
  25. 25Capstone Lesson 27: Eval Harness with Fixture TasksBuild a test bench that runs a coding agent on a folder of fixed tasks, scores each one pass or fail with a deterministic check, and reports pass rates, speed and cost.Build · ~90 minutes
  26. 26Capstone Lesson 28: Observability with OTel GenAI Spans and Prometheus MetricsBuild tracing for an agent that records each model and tool call as a span in the OpenTelemetry format and exposes counters for calls, tokens and timing.Build · ~90 minutes
  27. 27Capstone Lesson 29: End-to-End Coding Agent on the HarnessThe track's payoff: connect the gates, sandbox, test bench and tracing into one coding agent that fixes a small bug in a multi-file project, using a scripted policy instead of a live model.Build · ~90 minutes
  28. 28BPE Tokenizer From ScratchBuild a byte pair encoding tokenizer, the step that turns text into numbered pieces a language model reads and turns those numbers back into the same text.Build · ~90 minutes
  29. 29Tokenized Dataset with Sliding WindowBuild the data feeder for training: turn text into token ids and cut them into fixed-length overlapping windows, each paired with the next-token targets the model learns from.Build · ~90 minutes
  30. 30Token and Positional EmbeddingsBuild the two lookup tables that turn token ids into vectors and add a signal for each token's position, since a transformer has no sense of word order by itself.Build · ~90 minutes
  31. 31Multi-Head Self-AttentionBuild the attention block that lets each token look at earlier tokens and decide which matter, using several heads in parallel and a mask that hides the future.Build · ~90 minutes
  32. 32Transformer Block from ScratchBuild one transformer block with layer norm, attention, an MLP and skip connections, in both pre-norm and post-norm layouts, and see which trains stably when stacked deep.Build · ~90 minutes
  33. 33GPT Model AssemblyAssemble a full GPT-2 sized model from the earlier parts, count its parameters, and generate text with temperature and top-k sampling.Build · ~90 minutes
  34. 34Training Loop and EvaluationBuild the training loop for the GPT model: AdamW optimizer, warmup and cosine learning rate, loss on held-out data, sample text at intervals, and a log you can plot.Build · ~90 minutes
  35. 35Loading Pretrained WeightsLoad published GPT-2 style weights from a safetensors file into the model built earlier, map each parameter name, and generate text to confirm the load worked.Build · ~90 minutes
  36. 36Capstone Lesson 38: Classifier Fine-Tuning by Head SwapTurn a pretrained language model into a two-class classifier, such as spam or not spam, by replacing its output head, and compare training only the head with training the whole model.Build · ~90 minutes
  37. 37Capstone Lesson 39: Instruction Tuning by Supervised Fine-TuningTeach a base model to follow instructions by training it on instruction and response pairs, counting the loss only on the response part.Build · ~90 minutes
  38. 38Capstone Lesson 40: Direct Preference Optimization from ScratchBuild direct preference optimization, which trains a model straight from pairs of preferred and rejected answers without a separate reward model.Build · ~90 minutes
  39. 39Capstone Lesson 41: Full Evaluation PipelineBuild one evaluation pipeline that runs several different tests on any trained model, uses a local AI judge for open-ended answers, and combines the results into a report.Build · ~90 minutes
  40. 40Large Corpus DownloaderBuild a downloader for training data that streams compressed files, decompresses on the fly, removes near-duplicate documents and resumes cleanly after a network drop.Build · ~90 minutes
  41. 41HDF5 Tokenized CorpusStore a tokenized training corpus in HDF5 files that grow as data is added, are split across shards and can be read in fast chunks by many data loader workers.Build · ~90 minutes
  42. 42Cosine LR with Linear WarmupBuild the learning rate schedule most language models use: a slow linear ramp up at the start, then a smooth cosine decline toward zero.Build · ~90 minutes
  43. 43Gradient Clipping and Mixed PrecisionAdd two training safety belts: cap the size of gradient updates so one bad batch cannot wreck the model, and use mixed precision with loss scaling to train faster without numbers overflowing.Build · ~90 minutes
  44. 44Gradient AccumulationTrain with a large effective batch on small hardware by running several small batches, adding up their gradients and updating the model only once per group.Build · ~90 minutes
  45. 45Checkpoint Save and ResumeSave the full state of a training run, including model, optimizer, schedule, step count and random seeds, in a way that a crash at any moment still leaves a valid file.Build · ~90 minutes
  46. 46Distributed Data Parallel and FSDP from ScratchBuild multi-GPU training from scratch: copy starting weights to every worker, average gradients after each step, and shard model state so larger models fit in memory.Build · ~90 minutes
  47. 47Language Model Evaluation HarnessBuild an evaluation harness for language models: define each task, its metric and runner in one swappable shape, and rank models on a leaderboard.Build · ~90 minutes
  48. 48Hypothesis GeneratorThe first lesson of the research agent track: build a generator that drafts research ideas and forces each new draft to differ from the ones already made.Build · ~90 minutes
  49. 49Literature RetrievalBuild the search step that checks published work for a hypothesis before any experiment runs, so the agent does not waste effort on ideas already tested.Build · ~90 minutes
  50. 50Experiment RunnerBuild a runner that takes an experiment spec, runs it in a sandboxed subprocess with limits, and returns a JSON block of metrics the next step can trust.Build · ~90 minutes
  51. 51Result EvaluatorBuild the step that reads experiment metrics and decides whether a result is an improvement, a regression or just noise, ending in a one-line verdict.Build · ~90 minutes
  52. 52Paper WriterBuild a writer that fills a LaTeX paper template with the agent's results, so the document compiles or fails loudly instead of quietly coming out broken.Build · ~90 minutes
  53. 53Critic LoopBuild a critic that reviews drafts and returns feedback, engineered so it neither approves everything nor rejects everything, and the loop settles on a final version.Build · ~90 minutes
  54. 54Iteration SchedulerBuild the scheduler that decides which research directions to keep exploring and which to drop, within a fixed budget.Build · ~90 minutes
  55. 55End-to-End Research DemoThe track's payoff: connect the idea generator, literature check, runner, evaluator, writer, critic and scheduler into one research run that shows where any piece breaks.Build · ~90 minutes
  56. 56Vision Encoder PatchesBuild the first step of a vision model: cut an image into a grid of small squares, turn each square into a vector, and add a signal for where it sat in the image.Build · ~90 minutes
  57. 57Vision Transformer EncoderBuild a full vision transformer encoder that turns image patches into context-aware vectors, with a special token that summarizes the whole image.Build · ~90 minutes
  58. 58Projection Layer for Modality AlignmentBuild the small network that maps image vectors into the same space as text vectors, trained so a picture and its caption land close together.Build · ~90 minutes
  59. 59Cross-Attention FusionBuild cross-attention, where each text token looks at every image patch, so the model can tie each word it writes to a region of the image.Build · ~90 minutes
  60. 60Vision-Language PretrainingTrain the image encoder, projection and text decoder together with two goals: match images with their captions, and write captions for images.Build · ~90 minutes
  61. 61Multimodal EvaluationBuild three tests for vision-language models: image and caption retrieval scored by recall at 1, 5 and 10, visual question answering by exact match, and captioning by BLEU-4.Build · ~90 minutes
  62. 62Chunking Strategies, ComparedCompare ways to split documents into chunks for search, such as fixed size, by sentence, by structure or by meaning, and see how each affects what the system can find.Build · ~90 minutes
  63. 63Hybrid Retrieval with BM25 and Dense EmbeddingsCombine keyword search and meaning search, then merge their ranked lists with reciprocal rank fusion so each covers the queries the other misses.Build · ~90 minutes
  64. 64Cross-Encoder RerankerAdd a second-stage reranker that reads the question and each candidate document together and reorders the top results by how well they actually answer it.Build · ~90 minutes
  65. 65Query Rewriting: HyDE, Multi-Query, and DecompositionRewrite the user's question before searching: draft a likely answer and search with it, split a question into parts, or generate several versions of it.Build · ~90 minutes
  66. 66RAG Evaluation: Precision, Recall, MRR, nDCG, Faithfulness, Answer RelevanceBuild scoring for a document assistant that checks both the search step, with precision, recall, MRR and nDCG, and the answer step, with faithfulness and relevance.Build · ~90 minutes
  67. 67End-to-End RAG SystemThe track's payoff: combine chunking, hybrid search, reranking, query rewriting and evaluation into one document assistant with a built-in test loop.Build · ~90 minutes
  68. 68Task Spec FormatThe first lesson of the evaluation track: fix the file format for test tasks and the list of allowed metrics before writing any scoring code.Build · ~90 min
  69. 69Classical MetricsImplement the five classic text scores used in AI testing, BLEU, ROUGE-L, F1, exact match and accuracy, from scratch so you know what each number means.Build · ~90 min
  70. 70Code Exec MetricBuild a scorer for generated code that pulls the code out of the reply, runs it against tests in a protected process and reports honest pass rates.Build · ~90 min
  71. 71Perplexity and CalibrationMeasure two things: perplexity, how surprised a model is by new text, and calibration, whether its stated confidence matches how often it is right.Build · ~90 min
  72. 72Leaderboard AggregationCombine scores from many different tasks into a model ranking, and test whether gaps between models are statistically real or just chance.Build · ~90 min
  73. 73End-to-End Eval RunnerThe track's payoff: one runner that reads task files, calls any model through an adapter, scores the answers, adds calibration and produces a leaderboard.Build · ~90 min
  74. 74Collective Ops From ScratchThe first lesson of the distributed training track: build the four group operations that multi-GPU training relies on, allreduce, broadcast, allgather and reduce-scatter.Build · ~90 min
  75. 75Data Parallel DDP From ScratchBuild distributed data parallel training: start every worker with the same weights, and average each gradient across workers right after the backward pass.Build · ~90 min
  76. 76ZeRO Optimizer State ShardingBuild ZeRO stage 1, which splits the optimizer's memory across workers so each one stores only its share, then shares updated weights back after each step.Build · ~90 min
  77. 77Pipeline Parallel and Bubble AnalysisSplit a model across GPUs by layers, one stage per GPU, feed small micro-batches through like an assembly line, and measure the idle time at the start and end.Build · ~90 min
  78. 78Sharded Checkpoint and Atomic ResumeSave a checkpoint where each GPU writes its own shard in parallel, with a manifest that records who owns what, and resume as if nothing happened.Build · ~90 min
  79. 79End-to-End Distributed TrainingThe track's payoff: train a tiny GPT across four simulated workers with gradient syncing, optimizer sharding and a sharded checkpoint, and report loss and memory use.Build · ~90 min
  80. 80Capstone 82 — Jailbreak TaxonomyThe first lesson of the safety track: sort jailbreak attacks into named categories, such as role-play, encoding tricks and many-shot, so each can be tested and defended.Build · ~90 min
  81. 81Capstone 83 — Prompt Injection DetectorBuild a detector that reads a prompt and returns a confidence score and an attack category for prompt injection, instead of a vague yes or no.Build · ~90 min
  82. 82Capstone 84 — Refusal EvaluationMeasure two things separately: whether a model refuses harmful requests, and whether it still helps with harmless ones that only look risky.Build · ~90 min
  83. 83Capstone 85 — Content Classifier IntegrationAdd classifiers that check the model's output, and a policy router that combines them with input rules to decide whether to show, block or rewrite a reply.Build · ~90 min
  84. 84Capstone 86 — Constitutional Rules EngineBuild a rules engine where each rule has a name, a test and an explanation, so every block or allow decision can be traced to a specific written rule.Build · ~90 min
  85. 85Capstone 87 — End-to-End Safety GateThe track's payoff: one safety gate with checks before generation, during generation and after, producing one verdict and an audit record for every request.Build · ~90 min

Lessons from AI Engineering from Scratch by Rohit Ghumare and contributors, MIT license. Notes for owners by Lozo. License

Next step

Rather have it built for you?

Reading is optional. Tell me what's stuck and I'll build it into the business you already run.

Tell me what's stuck
  • A real conversation, not a pitch
  • You keep your accounts
  • Nothing sends without your yes