Lozo AI Course. AI engineering, with notes for owners.

1. 01Phases 7, 10

   How a model works

   A language model writes one word at a time, picking what most likely comes next from patterns in a huge pile of text. It has no memory between chats and sees nothing of your business unless you hand it over.

   Why it matters Know this and you stop expecting it to know your stock or last week's orders.

   Phase 7: Transformers Deep Dive →Phase 10: LLMs from Scratch →

   Sources the course links 2

   • PaperAttention Is All You Needarxiv.orgThe paper behind today's models. (opens in a new tab)
   • PaperLanguage Models are Few-Shot Learnersarxiv.orgThe GPT-3 paper: a model picks up a task from a few examples. (opens in a new tab)

2. 02Phase 11

   Prompting and context

   The prompt is your instructions. The context is everything the model can see when it answers: the prompt, your notes, a file, the last few messages.

   Why it matters Most weak answers come from thin context, not a weak model.

   Phase 11: LLM Engineering →

   Sources the course links 3

   • DocsPrompting best practicesplatform.claude.comAnthropic's guide: be direct, show examples. (opens in a new tab)
   • DocsPrompt engineeringdevelopers.openai.comOpenAI's guide to instructions and examples. (opens in a new tab)
   • PaperChain-of-Thought Promptingarxiv.orgAsking for the steps first improves hard answers. (opens in a new tab)

3. 03Phases 11, 13

   Tools and function calling

   A model can ask your software to do something: look up an order, book a slot, save a draft. You describe each tool, the model picks one and fills in the details, and your code runs it.

   Why it matters This is the step from a chat window to AI that does work inside your systems.

   Phase 11: LLM Engineering →Phase 13: Tools & Protocols →

   Sources the course links 3

   • DocsTool use with Claudeplatform.claude.comHow a model asks your code to act. (opens in a new tab)
   • DocsFunction callingdevelopers.openai.comThe same idea in OpenAI's API. (opens in a new tab)
   • DocsModel Context Protocolmodelcontextprotocol.ioAn open standard for plugging AI into your tools. (opens in a new tab)

4. 04Phase 11

   Knowledge bases (RAG)

   Retrieval-augmented generation: before the model answers, the system finds the right passages in your own documents and hands them over. The answer then rests on your policy, not a guess.

   Why it matters It is how an assistant answers from your handbook, menu or FAQ.

   Phase 11: LLM Engineering →

   Sources the course links 2

   • GuideContextual Retrievalanthropic.comAnthropic's method for finding the right passage more often. (opens in a new tab)
   • PaperRetrieval-Augmented Generationarxiv.orgThe paper that named the idea. (opens in a new tab)

5. 05Phase 14

   Agents

   An agent is a model in a loop: it plans, uses a tool, checks the result and goes again until the job is done or it needs you.

   Why it matters Agents fit open-ended jobs. When the steps never change, a fixed workflow is cheaper and safer.

   Phase 14: Agent Engineering →

   Sources the course links 3

   • GuideBuilding effective agentsanthropic.comWhen to use an agent, and when a workflow is enough. (opens in a new tab)
   • PaperReActarxiv.orgThink, act, look at the result: the loop most agents run. (opens in a new tab)
   • DocsOpenAI Agents SDKopenai.github.ioA small open toolkit for agents and handoffs. (opens in a new tab)

6. 06Phase 16

   Teams of agents

   Several agents split one job. A lead plans and hands parts to helpers that work side by side, then pulls their results together.

   Why it matters Worth it for wide research work. For a single task, one agent is simpler.

   Phase 16: Multi-Agent & Swarms →

   Sources the course links 3

   • GuideHow we built our multi-agent research systemanthropic.comAnthropic on a lead agent and its helpers. (opens in a new tab)
   • DocsAgent2Agent protocola2a-protocol.orgAn open standard for agents from different makers to work together. (opens in a new tab)
   • GuideOrchestrating agentsdevelopers.openai.comRoutines and handoffs, in plain code. (opens in a new tab)

7. 07Phase 11

   Evals

   An eval is a fixed set of real questions with the answers you expect. Run it each time you change a prompt, a model or a tool, and you see what got better or worse.

   Why it matters Without one, every change is a guess you test on customers.

   Phase 11: LLM Engineering →

   Sources the course links 3

   • PaperJudging LLM-as-a-Judgearxiv.orgUsing a strong model to grade answers, and where it slips. (opens in a new tab)
   • GuideThe evaluation guidebookhuggingface.coHugging Face's practical guide to testing models. (opens in a new tab)
   • Toolpromptfoopromptfoo.devAn open-source tool to test prompts side by side. (opens in a new tab)

8. 08Phases 11, 18

   Guardrails and safe use

   The checks around the model: what it may read, which actions wait for a person to approve, and filters on what goes in and what comes out.

   Why it matters One wrong action, like an email to the wrong list, undoes the time AI saved.

   Phase 11: LLM Engineering →Phase 18: Ethics, Safety & Alignment →

   Sources the course links 3

   • GuideOWASP Top 10 for LLM Applicationsgenai.owasp.orgThe common ways AI apps go wrong, and the fixes. (opens in a new tab)
   • DocsModerationdevelopers.openai.comOpenAI's free check for harmful text. (opens in a new tab)
   • RulesAI Actdigital-strategy.ec.europa.euThe EU's rules for AI, by level of risk. (opens in a new tab)

9. 09Phases 13, 17

   Shipping and running it

   Once it works, it has to stay fast, affordable and visible: reuse what repeats, log every call, and watch cost and errors week to week.

   Why it matters A build nobody watches drifts. The logs show what customers really ask.

   Phase 13: Tools & Protocols →Phase 17: Infrastructure & Production →

   Sources the course links 2

   • DocsPrompt cachingplatform.claude.comReuse the long part of a prompt for less. (opens in a new tab)
   • DocsOpenTelemetry for generative AIopentelemetry.ioA shared format for logging AI calls. (opens in a new tab)

1. 00Setup & Tooling0/13 13 lessons

   For owners. The engineer's workbench. Skim it so you know what a tidy, repeatable setup looks like when you hire one.

   1. 01Dev Environment
   2. 02Git & Collaboration
   3. 03GPU Setup & Cloud
   4. 04APIs & Keys
   5. 05Jupyter Notebooks
   6. 06Python Environments
   7. 07Docker for AI
   8. 08Editor Setup
   9. 09Data Management
   10. 10Terminal & Shell
   11. 11Linux for AI
   12. 12Debugging and Profiling
   13. 13Python for AI Engineering

   Open the phase: Setup & Tooling →
2. 01Math Foundations0/22 22 lessons

   For owners. The math under every model. You can skip it; the engineers you hire should know it.

   1. 01Linear Algebra Intuition
   2. 02Vectors, Matrices & Operations
   3. 03Matrix Transformations
   4. 04Calculus for Machine Learning
   5. 05Chain Rule & Automatic Differentiation
   6. 06Probability and Distributions
   7. 07Bayes' Theorem
   8. 08Optimization
   9. 09Information Theory
   10. 10Dimensionality Reduction
   11. 11Singular Value Decomposition
   12. 12Tensor Operations
   13. 13Numerical Stability
   14. 14Norms and Distances
   15. 15Statistics for Machine Learning
   16. 16Sampling Methods
   17. 17Linear Systems
   18. 18Convex Optimization
   19. 19Complex Numbers for AI
   20. 20The Fourier Transform
   21. 21Graph Theory for Machine Learning
   22. 22Stochastic Processes

   Open the phase: Math Foundations →
3. 02ML Fundamentals0/21 21 lessons

   For owners. Classic prediction, such as scoring leads or forecasting a busy week. Often simpler and cheaper than a chat model.

   1. 01What Is Machine Learning
   2. 02Linear Regression
   3. 03Logistic Regression
   4. 04Decision Trees and Random Forests
   5. 05Support Vector Machines
   6. 06K-Nearest Neighbors and Distances
   7. 07Unsupervised Learning
   8. 08Feature Engineering & Selection
   9. 09Model Evaluation
   10. 10Bias-Variance Tradeoff
   11. 11Ensemble Methods
   12. 12Hyperparameter Tuning
   13. 13ML Pipelines
   14. 14Naive Bayes
   15. 15Time Series Fundamentals
   16. 16Anomaly Detection
   17. 17Handling Imbalanced Data
   18. 18Feature Selection
   19. 19ML System Design — From Problem to Deployed Model
   20. 20Monitoring Models in Production — Data Drift and Concept Drift
   21. 21Feature Stores and Training-Serving Skew

   Open the phase: ML Fundamentals →
4. 03Deep Learning Core0/13 13 lessons

   For owners. How a neural network learns from examples. Background for everything after it; skim the notes.

   1. 01The Perceptron
   2. 02Multi-Layer Networks and Forward Pass
   3. 03Backpropagation from Scratch
   4. 04Activation Functions
   5. 05Loss Functions
   6. 06Optimizers
   7. 07Regularization
   8. 08Weight Initialization and Training Stability
   9. 09Learning Rate Schedules and Warmup
   10. 10Build Your Own Mini Framework
   11. 11Introduction to PyTorch
   12. 12Introduction to JAX
   13. 13Debugging Neural Networks

   Open the phase: Deep Learning Core →
5. 04Computer Vision0/28 28 lessons

   For owners. Software that reads images: receipts, shelf photos, damage photos, scanned forms.

   1. 01Image Fundamentals — Pixels, Channels, Color Spaces
   2. 02Convolutions from Scratch
   3. 03CNNs — LeNet to ResNet
   4. 04Image Classification
   5. 05Transfer Learning & Fine-Tuning
   6. 06Object Detection — YOLO from Scratch
   7. 07Semantic Segmentation — U-Net
   8. 08Instance Segmentation — Mask R-CNN
   9. 09Image Generation — GANs
   10. 10Image Generation — Diffusion Models
   11. 11Stable Diffusion — Architecture & Fine-Tuning
   12. 12Video Understanding — Temporal Modeling
   13. 133D Vision — Point Clouds & NeRFs
   14. 14Vision Transformers (ViT)
   15. 15Real-Time Vision — Edge Deployment
   16. 16Build a Complete Vision Pipeline — Capstone
   17. 17Self-Supervised Vision — SimCLR, DINO, MAE
   18. 18Open-Vocabulary Vision — CLIP
   19. 19OCR & Document Understanding
   20. 20Image Retrieval & Metric Learning
   21. 21Keypoint Detection & Pose Estimation
   22. 223D Gaussian Splatting from Scratch
   23. 23Diffusion Transformers & Rectified Flow
   24. 24SAM 3 & Open-Vocabulary Segmentation
   25. 25Vision-Language Models — The ViT-MLP-LLM Pattern
   26. 26Monocular Depth & Geometry Estimation
   27. 27Multi-Object Tracking & Video Memory
   28. 28World Models & Video Diffusion

   Open the phase: Computer Vision →
6. 05NLP, Foundations to Advanced0/29 29 lessons

   For owners. Working with text: sorting email, pulling details out of documents, reading reviews at scale.

   1. 01Text Processing — Tokenization, Stemming, Lemmatization
   2. 02Bag of Words, TF-IDF, and Text Representation
   3. 03Word Embeddings — Word2Vec from Scratch
   4. 04GloVe, FastText, and Subword Embeddings
   5. 05Sentiment Analysis
   6. 06Named Entity Recognition
   7. 07POS Tagging and Syntactic Parsing
   8. 08CNNs and RNNs for Text
   9. 09Sequence-to-Sequence Models
   10. 10Attention Mechanism — The Breakthrough
   11. 11Machine Translation
   12. 12Text Summarization
   13. 13Question Answering Systems
   14. 14Information Retrieval and Search
   15. 15Topic Modeling — LDA and BERTopic
   16. 16Text Generation Before Transformers — N-gram Language Models
   17. 17Chatbots — Rule-Based to Neural to LLM Agents
   18. 18Multilingual NLP
   19. 19Subword Tokenization — BPE, WordPiece, Unigram, SentencePiece
   20. 20Structured Outputs & Constrained Decoding
   21. 21Natural Language Inference — Textual Entailment
   22. 22Embedding Models — The 2026 Deep Dive
   23. 23Chunking Strategies for RAG
   24. 24Coreference Resolution
   25. 25Entity Linking & Disambiguation
   26. 26Relation Extraction & Knowledge Graph Construction
   27. 27LLM Evaluation — RAGAS, DeepEval, G-Eval
   28. 28Long-Context Evaluation — NIAH, RULER, LongBench, MRCR
   29. 29Dialogue State Tracking

   Open the phase: NLP, Foundations to Advanced →
7. 06Speech & Audio0/17 17 lessons

   For owners. Calls turned into text and text turned into a voice: the base of an AI phone line.

   1. 01Audio Fundamentals — Waveforms, Sampling, Fourier Transform
   2. 02Spectrograms, Mel Scale & Audio Features
   3. 03Audio Classification — From k-NN on MFCCs to AST and BEATs
   4. 04Speech Recognition (ASR) — CTC, RNN-T, Attention
   5. 05Whisper — Architecture & Fine-Tuning
   6. 06Speaker Recognition & Verification
   7. 07Text-to-Speech (TTS) — From Tacotron to F5 and Kokoro
   8. 08Voice Cloning & Voice Conversion
   9. 09Music Generation — MusicGen, Stable Audio, Suno, and the Licensing Earthquake
   10. 10Audio-Language Models — Qwen2.5-Omni, Audio Flamingo, GPT-4o Audio
   11. 11Real-Time Audio Processing
   12. 12Build a Voice Assistant Pipeline — The Phase 6 Capstone
   13. 13Neural Audio Codecs — EnCodec, SNAC, Mimi, DAC and the Semantic-Acoustic Split
   14. 14Voice Activity Detection & Turn-Taking — Silero, Cobra, and the Flush Trick
   15. 15Streaming Speech-to-Speech — Moshi, Hibiki, and Full-Duplex Dialogue
   16. 16Voice Anti-Spoofing & Audio Watermarking — ASVspoof 5, AudioSeal, WaveVerify
   17. 17Audio Evaluation — WER, MOS, UTMOS, MMAU, FAD, and the Open Leaderboards

   Open the phase: Speech & Audio →
8. 07Transformers Deep Dive0/16 16 lessons

   For owners. The design inside today's chat models. The owner notes say what that means for what a model can do.

   1. 01Why Transformers — The Problems with RNNs
   2. 02Self-Attention from Scratch
   3. 03Multi-Head Attention
   4. 04Positional Encoding — Sinusoidal, RoPE, ALiBi
   5. 05The Full Transformer — Encoder + Decoder
   6. 06BERT — Masked Language Modeling
   7. 07GPT — Causal Language Modeling
   8. 08T5, BART — Encoder-Decoder Models
   9. 09Vision Transformers (ViT)
   10. 10Audio Transformers — Whisper Architecture
   11. 11Mixture of Experts (MoE)
   12. 12KV Cache, Flash Attention & Inference Optimization
   13. 13Scaling Laws
   14. 14Build a Transformer from Scratch — The Capstone
   15. 15Attention Variants — Sliding Window, Sparse, Differential
   16. 16Speculative Decoding — Draft, Verify, Repeat

   Open the phase: Transformers Deep Dive →
9. 08Generative AI0/15 15 lessons

   For owners. Making images, audio and video. Handy for drafts, with a person checking each one before it goes out.

   1. 01Generative Models — Taxonomy & History
   2. 02Autoencoders & Variational Autoencoders (VAE)
   3. 03GANs — Generator vs Discriminator
   4. 04Conditional GANs & Pix2Pix
   5. 05StyleGAN
   6. 06Diffusion Models — DDPM from Scratch
   7. 07Latent Diffusion & Stable Diffusion
   8. 08ControlNet, LoRA & Conditioning
   9. 09Inpainting, Outpainting & Image Editing
   10. 10Video Generation
   11. 11Audio Generation
   12. 123D Generation
   13. 13Flow Matching & Rectified Flows
   14. 14Evaluation — FID, CLIP Score, Human Preference
   15. 15Visual Autoregressive Modeling (VAR): Next-Scale Prediction

   Open the phase: Generative AI →
10. 09Reinforcement Learning0/12 12 lessons

    For owners. How a model learns from feedback and rewards. Mostly for engineers.

    1. 01MDPs, States, Actions & Rewards
    2. 02Dynamic Programming — Policy Iteration & Value Iteration
    3. 03Monte Carlo Methods — Learning from Complete Episodes
    4. 04Temporal Difference — Q-Learning & SARSA
    5. 05Deep Q-Networks (DQN)
    6. 06Policy Gradient — REINFORCE from Scratch
    7. 07Actor-Critic — A2C and A3C
    8. 08Proximal Policy Optimization (PPO)
    9. 09Reward Modeling & RLHF
    10. 10Multi-Agent RL
    11. 11Sim-to-Real Transfer
    12. 12RL for Games — AlphaZero, MuZero, and the LLM-Reasoning Era

    Open the phase: Reinforcement Learning →
11. 10LLMs from Scratch0/24 24 lessons

    For owners. Building a language model end to end. It shows why a model knows nothing of your business until you hand it over.

    1. 01Tokenizers: BPE, WordPiece, SentencePiece
    2. 02Building a Tokenizer from Scratch
    3. 03Data Pipelines for Pre-Training
    4. 04Pre-Training a Mini GPT (124M Parameters)
    5. 05Scaling: Distributed Training, FSDP, DeepSpeed
    6. 06Instruction Tuning (SFT)
    7. 07RLHF: Reward Model + PPO
    8. 08DPO: Direct Preference Optimization
    9. 09Constitutional AI and Self-Improvement
    10. 10Evaluation: Benchmarks, Evals, LM Harness
    11. 11Quantization: Making Models Fit
    12. 12Inference Optimization
    13. 13Building a Complete LLM Pipeline
    14. 14Open Models: Architecture Walkthroughs
    15. 15Speculative Decoding and EAGLE-3
    16. 16Differential Attention (V2)
    17. 17Native Sparse Attention (DeepSeek NSA)
    18. 18Multi-Token Prediction (MTP)
    19. 19DualPipe Parallelism
    20. 20DeepSeek-V3 Architecture Walkthrough
    21. 21Jamba — Hybrid SSM-Transformer
    22. 22Async and Hogwild! Inference
    23. 23Speculative Decoding and EAGLE
    24. 24Gradient Checkpointing and Activation Recomputation

    Open the phase: LLMs from Scratch →
12. 11LLM Engineering0/17 17 lessons

    For owners. Putting models to work: prompts, knowledge bases, evals and guardrails. The phase most worth an owner's time.

    1. 01Prompt Engineering: Techniques & Patterns
    2. 02Few-Shot, Chain-of-Thought, Tree-of-Thought
    3. 03Structured Outputs: JSON, Schema Validation, Constrained Decoding
    4. 04Embeddings & Vector Representations
    5. 05Context Engineering: Windows, Budgets, Memory, and Retrieval
    6. 06RAG (Retrieval-Augmented Generation)
    7. 07Advanced RAG (Chunking, Reranking, Hybrid Search)
    8. 08Fine-Tuning with LoRA & QLoRA
    9. 09Function Calling & Tool Use
    10. 10Evaluation & Testing LLM Applications
    11. 11Caching, Rate Limiting & Cost Optimization
    12. 12Guardrails, Safety & Content Filtering
    13. 13Building a Production LLM Application
    14. 14Model Context Protocol (MCP)
    15. 15Prompt Caching and Context Caching
    16. 16Agent State Machines — Graphs, Nodes, Checkpoints
    17. 17Agent Framework Tradeoffs — Graph, Role, and Actor Orchestration

    Open the phase: LLM Engineering →
13. 12Multimodal AI0/25 25 lessons

    For owners. Models that take a photo, a voice note and text together, such as reading a photo of an invoice.

    1. 01Vision Transformers and the Patch-Token Primitive
    2. 02CLIP and Contrastive Vision-Language Pretraining
    3. 03From CLIP to BLIP-2 — Q-Former as Modality Bridge
    4. 04Flamingo and Gated Cross-Attention for Few-Shot VLMs
    5. 05LLaVA and Visual Instruction Tuning
    6. 06Any-Resolution Vision: Patch-n'-Pack and NaFlex
    7. 07Open-Weight VLM Recipes: What Actually Matters
    8. 08LLaVA-OneVision: Single-Image, Multi-Image, Video in One Model
    9. 09Qwen-VL Family and Dynamic-FPS Video
    10. 10InternVL3: Native Multimodal Pretraining
    11. 11Chameleon and Early-Fusion Token-Only Multimodal Models
    12. 12Emu3: Next-Token Prediction for Image and Video Generation
    13. 13Transfusion: Autoregressive Text + Diffusion Image in One Transformer
    14. 14Show-o and Discrete-Diffusion Unified Models
    15. 15Janus-Pro: Decoupled Encoders for Unified Multimodal Models
    16. 16MIO and Any-to-Any Streaming Multimodal Models
    17. 17Video-Language Models: Temporal Tokens and Grounding
    18. 18Long-Video Understanding at Million-Token Context
    19. 19Audio-Language Models: the Whisper to Audio Flamingo 3 Arc
    20. 20Omni Models: Qwen2.5-Omni and the Thinker-Talker Split
    21. 21Embodied VLAs: RT-2, OpenVLA, π0, GR00T
    22. 22Document and Diagram Understanding
    23. 23ColPali and Vision-Native Document RAG
    24. 24Multimodal RAG and Cross-Modal Retrieval
    25. 25Multimodal Agents and Computer-Use (Capstone)

    Open the phase: Multimodal AI →
14. 13Tools & Protocols0/45 45 lessons

    For owners. How AI connects to the tools you already use: the calendar, the inbox, the order system.

    1. 01The Tool Interface — Why Agents Need Structured I/O
    2. 02Function Calling Deep Dive — OpenAI, Anthropic, Gemini
    3. 03Parallel Tool Calls and Streaming with Tools
    4. 04Structured Output — JSON Schema, Pydantic, Zod, Constrained Decoding
    5. 05Tool Schema Design — Naming, Descriptions, Parameter Constraints
    6. 06MCP Fundamentals: Stateless Requests and JSON-RPC
    7. 07Building an MCP Server: Stateless Python and TypeScript
    8. 08Building an MCP Client: Discovery, Routing, and Dual-Era Fallback
    9. 09MCP Transports: stdio and Stateless Streamable HTTP
    10. 10MCP Resources and Prompts: Addressable Context for Stateless Servers
    11. 11MCP Model Input: Sampling Migration and Stateless MRTR
    12. 12Explicit Scope and Stateless Elicitation
    13. 13MCP Tasks Extension: Durable Work on a Stateless Core
    14. 14MCP Apps on the Stateless Protocol
    15. 15MCP Security: Poisoned Metadata, Routing, and MRTR State
    16. 16MCP Authorization: CIMD, Issuer Binding, PKCE, and Step-Up
    17. 17Stateless MCP Gateways and Registry Admission
    18. 18MCP Auth in Production: Issuer-Bound Enrollment and Tokens
    19. 19A2A — Agent-to-Agent Protocol
    20. 20OpenTelemetry GenAI — Tracing Tool Calls End-to-End
    21. 21LLM Routing Layer — LiteLLM, OpenRouter, Portkey
    22. 22Agent Skills: Portable Contract and Runtime Boundary
    23. 23Capstone: Stateless Tool Ecosystem
    24. 24Skill Discovery and Progressive Disclosure
    25. 25Skill Invocation and Routing
    26. 26Skill Permissions, Sandboxes, and Trust
    27. 27Skill Evals, Packaging, and Portability
    28. 28MCP Tool Contracts and Content
    29. 29MCP Reliability, Cancellation, and Flow Control
    30. 30MCP Registry Supply Chain: Admission, Drift, and Rollback
    31. 31MCP Conformance Engineering: Versioning, Evidence, and Operations
    32. 32Program Anatomy: An ELF Reader From Scratch
    33. 33Disassembly and Control Flow: From Bytes to a Graph You Can Check
    34. 34Lifting to Pseudocode: IR, SSA, and Structuring From Scratch
    35. 35Evidence, Confidence, and Unknowns in Binary Analysis
    36. 36Rebuild and Prove Equal: Differential Tests, Byte Comparison, and the Evidence Loop
    37. 37The Reverse Engineering Landscape: History, Tool Families, and How a Decompiler Is Built
    38. 38Calling Conventions and Stack Frames: System V, Windows x64, and AArch64
    39. 39Compiler Idioms: What Optimization Does to Your Code and How to Read It Back
    40. 40Linking, Loading, and Stripped Binaries: PLT, GOT, Imports, Fixups, and Function Identification
    41. 41Recovering Data and Types: Structs, Arrays, C++ Objects, and DWARF
    42. 42Dynamic Analysis of Your Own Program: Debuggers, Traces, and Confirming Static Claims
    43. 43Beyond Native Code: Python, JVM, .NET, and WebAssembly Bytecode
    44. 44Binary Diffing and Symbolic Execution: Matching Functions and Solving for Inputs
    45. 45AI-Assisted Reverse Engineering: LLM Decompilers, Analysis Agents, and Ground Truth

    Open the phase: Tools & Protocols →
15. 14Agent Engineering0/54 54 lessons

    For owners. Agents that take steps on their own. Read the notes before you let one act for your business.

    1. 01The Agent Loop: Observe, Think, Act
    2. 02ReWOO and Plan-and-Execute: Decoupled Planning
    3. 03Reflexion: Verbal Reinforcement Learning
    4. 04Tree of Thoughts and LATS: Deliberate Search
    5. 05Self-Refine and CRITIC: Iterative Output Improvement
    6. 06Tool Use and Function Calling
    7. 07Agent Memory — Virtual Context and Memory Paging
    8. 08Memory Blocks and Sleep-Time Compute
    9. 09Hybrid Memory: Vector + Graph + KV
    10. 10Skill Libraries and Lifelong Learning (Voyager)
    11. 11Planning with HTN and Evolutionary Search
    12. 12Anthropic's Workflow Patterns: Simple Over Complex
    13. 13Stateful Graph Orchestration — Durable Execution and Checkpoints
    14. 14The Actor Model for Agents — Async Messages and Typed Runtimes
    15. 15Role-Based Agent Teams — Roles, Tasks, Processes
    16. 16OpenAI Agents SDK: Handoffs, Guardrails, Tracing
    17. 17The Harness as a Library — Subagents and Session Store
    18. 18Production Agent Runtimes — Fast Instantiation and Typed Workflows
    19. 19Benchmarks: SWE-bench, GAIA, AgentBench
    20. 20Benchmarks: WebArena and OSWorld
    21. 21Computer Use: Claude, OpenAI CUA, Gemini
    22. 22Voice Agents: Pipecat and LiveKit
    23. 23OpenTelemetry GenAI Semantic Conventions
    24. 24Agent Observability: Langfuse, Phoenix, Opik
    25. 25Multi-Agent Debate and Collaboration
    26. 26Failure Modes: Why Agents Break
    27. 27Prompt Injection and the PVE Defense
    28. 28Orchestration Patterns: Supervisor, Swarm, Hierarchical
    29. 29Production Runtimes: Queue, Event, Cron
    30. 30Eval-Driven Agent Development
    31. 31Agent Workbench Engineering: Why Capable Models Still Fail
    32. 32The Minimal Agent Workbench
    33. 33Agent Instructions as Executable Constraints
    34. 34Repo Memory and Durable State
    35. 35Initialization Scripts for Agents
    36. 36Scope Contracts and Task Boundaries
    37. 37Runtime Feedback Loops
    38. 38Verification Gates
    39. 39Reviewer Agent: Separate Builder from Marker
    40. 40Multi-Session Handoff
    41. 41The Workbench on a Real Repo
    42. 42Capstone: Ship a Reusable Agent Workbench Pack
    43. 43Frame the Task Before the Agent Writes Code
    44. 44Build an Evidence-Backed Execution Plan
    45. 45Delegate Agent Work with Isolation and Merge Contracts
    46. 46Turn Every Agent Correction into a System Improvement
    47. 47Define the Outcome Before You Choose the Output
    48. 48Discover the Workflow People Actually Perform
    49. 49Map Assumptions and Resolve the Riskiest One First
    50. 50Choose the Smallest Slice That Can Change the Decision
    51. 51Write Specifications That Preserve Judgment
    52. 52Design Success Metrics Before the Result Exists
    53. 53Choose Prototype, Pilot, or Production Deliberately
    54. 54Build a Feedback Ratchet with Ownership and Retirement

    Open the phase: Agent Engineering →
16. 15Autonomous Systems0/22 22 lessons

    For owners. Agents that run without a person watching. The limits matter more than the features.

    1. 01The Shift from Chatbots to Long-Horizon Agents
    2. 02STaR, V-STaR, Quiet-STaR — Self-Taught Reasoning
    3. 03AlphaEvolve — Evolutionary Coding Agents
    4. 04Darwin Godel Machine — Open-Ended Self-Modifying Agents
    5. 05AI Scientist v2 — Workshop-Level Autonomous Research
    6. 06Automated Alignment Research (Anthropic AAR)
    7. 07Recursive Self-Improvement — Capability vs Alignment
    8. 08Bounded Self-Improvement Designs
    9. 09The Autonomous Coding Agent Landscape (2026)
    10. 10Permission Modes for Autonomous Agents
    11. 11Browser Agents and Long-Horizon Web Tasks
    12. 12Long-Running Background Agents: Durable Execution
    13. 13Action Budgets, Iteration Caps, and Cost Governors
    14. 14Kill Switches, Circuit Breakers, and Canary Tokens
    15. 15Human-in-the-Loop: Propose-Then-Commit
    16. 16Checkpoints and Rollback
    17. 17Constitutional AI and Rule Overrides
    18. 18Llama Guard and Input/Output Classification
    19. 19Anthropic Responsible Scaling Policy v3.0
    20. 20OpenAI Preparedness Framework and DeepMind Frontier Safety Framework
    21. 21METR Time Horizons and External Capability Evaluation
    22. 22CAIS, CAISI, and Societal-Scale Risk

    Open the phase: Autonomous Systems →
17. 16Multi-Agent & Swarms0/25 25 lessons

    For owners. Several agents splitting one job. Worth it for big jobs, with more parts to check.

    1. 01Why Multi-Agent?
    2. 02Heritage of FIPA-ACL and Speech Acts
    3. 03Communication Protocols
    4. 04The Multi-Agent Primitive Model
    5. 05Supervisor / Orchestrator-Worker Pattern
    6. 06Hierarchical Architecture and Its Failure Mode
    7. 07Society of Mind and Multi-Agent Debate
    8. 08Role Specialization — Planner, Critic, Executor, Verifier
    9. 09Parallel / Swarm / Networked Architectures
    10. 10Group Chat and Speaker Selection
    11. 11Handoffs and Routines — Stateless Orchestration
    12. 12A2A — The Agent-to-Agent Protocol
    13. 13Shared Memory and Blackboard Patterns
    14. 14Consensus and Byzantine Fault Tolerance for Agents
    15. 15Voting, Self-Consistency, and Debate Topology
    16. 16Negotiation and Bargaining
    17. 17Generative Agents and Emergent Simulation
    18. 18Theory of Mind and Emergent Coordination
    19. 19Swarm Optimization for LLMs (PSO, ACO)
    20. 20MARL — MADDPG, QMIX, MAPPO
    21. 21Agent Economies, Token Incentives, Reputation
    22. 22Production Scaling — Queues, Checkpoints, Durability
    23. 23Failure Modes — MAST, Groupthink, Monoculture, Cascading Errors
    24. 24Evaluation and Coordination Benchmarks
    25. 25Case Studies and the 2026 State of the Art

    Open the phase: Multi-Agent & Swarms →
18. 17Infrastructure & Production0/29 29 lessons

    For owners. Keeping an AI build fast, affordable and watched once it is live.

    1. 01Managed LLM Platforms — Bedrock, Vertex AI, Azure OpenAI
    2. 02Inference Platform Economics — Fireworks, Together, Baseten, Modal, Replicate, Anyscale
    3. 03GPU Autoscaling on Kubernetes — Karpenter, KAI Scheduler, Gang Scheduling
    4. 04Serving Engine Internals — PagedAttention, Continuous Batching, Chunked Prefill
    5. 05EAGLE-3 Speculative Decoding in Production
    6. 06Prefix-Cache Serving — RadixAttention and KV Reuse
    7. 07Hardware-Specialized Inference Compilation — FP8 and NVFP4 on Blackwell
    8. 08Inference Metrics — TTFT, TPOT, ITL, Goodput, P99
    9. 09Production Quantization — AWQ, GPTQ, GGUF K-quants, FP8, MXFP4/NVFP4
    10. 10Cold Start Mitigation for Serverless LLMs
    11. 11Multi-Region LLM Serving and KV Cache Locality
    12. 12Edge Inference — Apple Neural Engine, Qualcomm Hexagon, WebGPU/WebLLM, Jetson
    13. 13LLM Observability Stack Selection
    14. 14Prompt Caching and Semantic Caching Economics
    15. 15Batch APIs — the 50% Discount as Industry Standard
    16. 16Model Routing as a Cost-Reduction Primitive
    17. 17Disaggregated Prefill/Decode — NVIDIA Dynamo and llm-d
    18. 18Production Serving Stack — KV Offloading and Cache-Aware Routing
    19. 19AI Gateways — LiteLLM, Portkey, Kong AI Gateway, Bifrost
    20. 20Shadow Traffic, Canary Rollout, and Progressive Deployment for LLMs
    21. 21A/B Testing LLM Features — GrowthBook, Statsig, and the Vibes Problem
    22. 22Load Testing LLM APIs — Why k6 and Locust Lie
    23. 23SRE for AI — Multi-Agent Incident Response, Runbooks, Predictive Detection
    24. 24Chaos Engineering for LLM Production
    25. 25Security — Secrets, API Key Rotation, Audit Logs, Guardrails
    26. 26Compliance — SOC 2, HIPAA, GDPR, PCI-DSS, EU AI Act, ISO 42001
    27. 27FinOps for LLMs — Unit Economics and Multi-Tenant Attribution
    28. 28Self-Hosted Serving Selection — Matching Engine to Hardware and Scale
    29. 29Model Demo Interfaces — From a Function to a Shareable App

    Open the phase: Infrastructure & Production →
19. 18Ethics, Safety & Alignment0/30 30 lessons

    For owners. Using AI safely: privacy, fairness, and where a person has to decide.

    1. 01Instruction-Following as Alignment Signal
    2. 02Reward Hacking and Goodhart's Law
    3. 03The Direct Preference Optimization Family
    4. 04Sycophancy as RLHF Amplification
    5. 05Constitutional AI and RLAIF
    6. 06Mesa-Optimization and Deceptive Alignment
    7. 07Sleeper Agents — Persistent Deception
    8. 08In-Context Scheming in Frontier Models
    9. 09Alignment Faking
    10. 10AI Control — Safety Despite Subversion
    11. 11Scalable Oversight and Weak-to-Strong Generalization
    12. 12Red-Teaming: PAIR and Automated Attacks
    13. 13Many-Shot Jailbreaking
    14. 14ASCII Art and Visual Jailbreaks
    15. 15Indirect Prompt Injection — Production Attack Surface
    16. 16Red-Team Tooling — Garak, Llama Guard, PyRIT
    17. 17WMDP and Dual-Use Capability Evaluation
    18. 18Frontier Safety Frameworks — RSP, PF, FSF
    19. 19Anthropic's Model Welfare Program
    20. 20Bias and Representational Harm in LLMs
    21. 21Fairness Criteria — Group, Individual, Counterfactual
    22. 22Differential Privacy for LLMs
    23. 23Watermarking — SynthID, Stable Signature, C2PA
    24. 24Regulatory Frameworks — EU, US, UK, Korea
    25. 25EchoLeak and the Emergence of CVEs for AI
    26. 26Model, System, and Dataset Cards
    27. 27Data Provenance and Training-Data Governance
    28. 28Alignment Research Ecosystem — MATS, Redwood, Apollo, METR
    29. 29Moderation Systems — OpenAI, Perspective, Llama Guard
    30. 30Dual-Use Risk — Cyber, Bio, Chem, Nuclear Uplift

    Open the phase: Ethics, Safety & Alignment →
20. 19Capstone Projects0/85 85 lessons

    For owners. Full projects that pull the course together. A good test for an engineer you plan to hire.

    1. 01Capstone 01 — Terminal-Native Coding Agent
    2. 02Capstone 02 — RAG over Codebase (Cross-Repo Semantic Search)
    3. 03Capstone 03 — Real-Time Voice Assistant (ASR to LLM to TTS)
    4. 04Capstone 04 — Multimodal Document QA (Vision-First PDF, Tables, Charts)
    5. 05Capstone 05 — Autonomous Research Agent (AI-Scientist Class)
    6. 06Capstone 06 — DevOps Troubleshooting Agent for Kubernetes
    7. 07Capstone 07 — End-to-End Fine-Tuning Pipeline (Data to SFT to DPO to Serve)
    8. 08Capstone 08 — Production RAG Chatbot for a Regulated Vertical
    9. 09Capstone 09 — Code Migration Agent (Repo-Level Language / Runtime Upgrade)
    10. 10Capstone 10 — Multi-Agent Software Engineering Team
    11. 11Capstone 11 — LLM Observability & Eval Dashboard
    12. 12Capstone 12 — Video Understanding Pipeline (Scene, QA, Search)
    13. 13Capstone 13: Stateless MCP Server with Registry and Governance
    14. 14Capstone 14 — Speculative-Decoding Inference Server
    15. 15Capstone 15 — Constitutional Safety Harness + Red-Team Range
    16. 16Capstone 16 — GitHub Issue-to-PR Autonomous Agent
    17. 17Capstone 17 — Personal AI Tutor (Adaptive, Multimodal, with Memory)
    18. 18Agent Harness Loop Contract
    19. 19Tool Registry with Schema Validation
    20. 20JSON-RPC 2.0 Over Newline-Delimited Stdio
    21. 21Function Call Dispatcher
    22. 22Plan-Execute Control Flow
    23. 23Capstone Lesson 25: Verification Gates and the Observation Budget
    24. 24Capstone Lesson 26: Sandbox Runner with Denylist and Path Jail
    25. 25Capstone Lesson 27: Eval Harness with Fixture Tasks
    26. 26Capstone Lesson 28: Observability with OTel GenAI Spans and Prometheus Metrics
    27. 27Capstone Lesson 29: End-to-End Coding Agent on the Harness
    28. 28BPE Tokenizer From Scratch
    29. 29Tokenized Dataset with Sliding Window
    30. 30Token and Positional Embeddings
    31. 31Multi-Head Self-Attention
    32. 32Transformer Block from Scratch
    33. 33GPT Model Assembly
    34. 34Training Loop and Evaluation
    35. 35Loading Pretrained Weights
    36. 36Capstone Lesson 38: Classifier Fine-Tuning by Head Swap
    37. 37Capstone Lesson 39: Instruction Tuning by Supervised Fine-Tuning
    38. 38Capstone Lesson 40: Direct Preference Optimization from Scratch
    39. 39Capstone Lesson 41: Full Evaluation Pipeline
    40. 40Large Corpus Downloader
    41. 41HDF5 Tokenized Corpus
    42. 42Cosine LR with Linear Warmup
    43. 43Gradient Clipping and Mixed Precision
    44. 44Gradient Accumulation
    45. 45Checkpoint Save and Resume
    46. 46Distributed Data Parallel and FSDP from Scratch
    47. 47Language Model Evaluation Harness
    48. 48Hypothesis Generator
    49. 49Literature Retrieval
    50. 50Experiment Runner
    51. 51Result Evaluator
    52. 52Paper Writer
    53. 53Critic Loop
    54. 54Iteration Scheduler
    55. 55End-to-End Research Demo
    56. 56Vision Encoder Patches
    57. 57Vision Transformer Encoder
    58. 58Projection Layer for Modality Alignment
    59. 59Cross-Attention Fusion
    60. 60Vision-Language Pretraining
    61. 61Multimodal Evaluation
    62. 62Chunking Strategies, Compared
    63. 63Hybrid Retrieval with BM25 and Dense Embeddings
    64. 64Cross-Encoder Reranker
    65. 65Query Rewriting: HyDE, Multi-Query, and Decomposition
    66. 66RAG Evaluation: Precision, Recall, MRR, nDCG, Faithfulness, Answer Relevance
    67. 67End-to-End RAG System
    68. 68Task Spec Format
    69. 69Classical Metrics
    70. 70Code Exec Metric
    71. 71Perplexity and Calibration
    72. 72Leaderboard Aggregation
    73. 73End-to-End Eval Runner
    74. 74Collective Ops From Scratch
    75. 75Data Parallel DDP From Scratch
    76. 76ZeRO Optimizer State Sharding
    77. 77Pipeline Parallel and Bubble Analysis
    78. 78Sharded Checkpoint and Atomic Resume
    79. 79End-to-End Distributed Training
    80. 80Capstone 82 — Jailbreak Taxonomy
    81. 81Capstone 83 — Prompt Injection Detector
    82. 82Capstone 84 — Refusal Evaluation
    83. 83Capstone 85 — Content Classifier Integration
    84. 84Capstone 86 — Constitutional Rules Engine
    85. 85Capstone 87 — End-to-End Safety Gate

    Open the phase: Capstone Projects →

Lessons from AI Engineering from Scratch by Rohit Ghumare and contributors, MIT license. Notes for owners by Lozo. License