Infrastructure & Production
For ownersKeeping an AI build fast, affordable and watched once it is live.
Ship AI to the real world. Scale, monitor, optimize.
The lessons
- 01Managed LLM Platforms — Bedrock, Vertex AI, Azure OpenAIThe big cloud companies each rent out AI models through their own platform. AWS Bedrock offers many model makers in one place, Azure OpenAI centers on OpenAI models, and Vertex AI centers on Gemini.Learn · ~60 minutes
- 02Inference Platform Economics — Fireworks, Together, Baseten, Modal, Replicate, AnyscaleBeyond the big clouds, a set of companies such as Fireworks, Together, Baseten, Modal and Replicate host open models for you. Each one is strongest at something different: speed, model choice, enterprise support, developer ease or image and video models.Learn · ~60 minutes
- 03GPU Autoscaling on Kubernetes — Karpenter, KAI Scheduler, Gang SchedulingThis lesson covers how a company running its own AI models adds and removes graphics chips automatically as traffic changes, and how it avoids jobs that wait forever because only part of the hardware they need is free.Learn · ~75 minutes
- 04Serving Engine Internals — PagedAttention, Continuous Batching, Chunked PrefillThis explains the main tricks that let one server answer many AI requests at the same time: storing model memory in small pages, adding new requests into a running batch, and splitting long prompts so short answers are not stuck behind them.Learn · ~75 minutes
- 05EAGLE-3 Speculative Decoding in ProductionSpeculative decoding speeds up AI answers by letting a small, fast helper guess the next few words while the big model checks them all at once. EAGLE-3 is a newer version where the helper learns from the big model's own inner signals.Learn · ~60 minutes
- 06Prefix-Cache Serving — RadixAttention and KV ReuseWhen many requests start with the same text, such as one long instruction block, the server can reuse the work it already did on that shared start. SGLang organizes this saved work in a tree and schedules requests to reuse it.Learn · ~75 minutes
- 07Hardware-Specialized Inference Compilation — FP8 and NVFP4 on BlackwellTensorRT-LLM is NVIDIA software that rebuilds a model to run as fast as possible on NVIDIA chips, often using smaller number formats like FP8 and FP4. It trades flexibility for speed on that one hardware family.Learn · ~75 minutes
- 08Inference Metrics — TTFT, TPOT, ITL, Goodput, P99These are the measures that tell you if an AI service feels fast: time until the first word appears, time between later words, total time, and goodput, which counts only the requests that met your speed target.Learn · ~60 minutes
- 09Production Quantization — AWQ, GPTQ, GGUF K-quants, FP8, MXFP4/NVFP4Quantization stores a model's numbers in fewer bits so it uses less memory and runs faster. The lesson compares the common formats and explains that the right one depends on your hardware and serving software.Learn · ~75 minutes
- 10Cold Start Mitigation for Serverless LLMsA large model can take minutes to load onto a fresh machine before it answers anything. This lesson covers ways to cut that delay, such as preloaded machine images, streaming the model in, memory snapshots and keeping some servers warm.Learn · ~60 minutes
- 11Multi-Region LLM Serving and KV Cache LocalityWhen AI servers run in several places, sending each request to the server that already holds its saved prompt work makes answers much faster than spreading requests evenly. This lesson covers routers that choose servers by cache match.Learn · ~60 minutes
- 12Edge Inference — Apple Neural Engine, Qualcomm Hexagon, WebGPU/WebLLM, JetsonEdge inference means running AI models on the device itself, such as a phone, laptop, browser or small computer, instead of in the cloud. The main limit is how fast the device can move data through its memory.Learn · ~60 minutes
- 13LLM Observability Stack SelectionObservability tools record what your AI system does on every request: the prompt, the answer, the cost, the time taken and any errors. Some also bundle testing and prompt management. The lesson shows how to choose among them.Learn · ~60 minutes
- 14Prompt Caching and Semantic Caching EconomicsCaching saves money by reusing work. Prompt caching lets the provider reuse a repeated prompt beginning at a lower price. Semantic caching returns a saved answer when a new question means the same thing as an earlier one.Learn · ~60 minutes
- 15Batch APIs — the 50% Discount as Industry StandardBatch APIs let you send many AI requests at once and get the results back later, often within a day, at a discount compared with instant calls. Most major model providers offer this option.Learn · ~45 minutes
- 16Model Routing as a Cost-Reduction PrimitiveModel routing sends each request to the cheapest model that can handle it, and only passes hard ones to an expensive top model. A small check at the front decides which way each request goes.Learn · ~60 minutes
- 17Disaggregated Prefill/Decode — NVIDIA Dynamo and llm-dReading the prompt and writing the answer stress a GPU in different ways. Disaggregation puts these two steps on separate groups of GPUs and passes the working memory between them, so each group is used fully.Learn · ~75 minutes
- 18Production Serving Stack — KV Offloading and Cache-Aware RoutingThis lesson shows a full production setup for self-hosted AI: a router, several serving engines and monitoring, all on Kubernetes. It adds LMCache, which moves saved prompt work off the GPU into regular memory or disk so it can be reused.Learn · ~60 minutes
- 19AI Gateways — LiteLLM, Portkey, Kong AI Gateway, BifrostAn AI gateway is a single middle layer between your apps and the model providers. It handles switching providers, retries, spending limits, keys, logs and safety filters in one place.Learn · ~60 minutes
- 20Shadow Traffic, Canary Rollout, and Progressive Deployment for LLMsChanging the model or prompt behind a live product is risky. The safer path is to run the new version quietly on copied traffic first, then send a small share of real users to it, then widen step by step.Learn · ~60 minutes
- 21A/B Testing LLM Features — GrowthBook, Statsig, and the Vibes ProblemA/B testing shows real users two versions of an AI feature and measures which one they prefer or act on. Offline tests show whether a model can do the job; A/B tests show whether people care.Learn · ~60 minutes
- 22Load Testing LLM APIs — Why k6 and Locust LieLoad testing checks how an AI service behaves when many people use it at once. Standard tools can give misleading numbers for streaming AI answers, so the lesson covers AI-aware methods and common traps.Build · ~75 minutes
- 23SRE for AI — Multi-Agent Incident Response, Runbooks, Predictive DetectionSite reliability engineering keeps systems running. AI SRE uses language models, fed with logs, runbooks and system maps, to help investigate outages, while humans still approve the important decisions.Learn · ~60 minutes
- 24Chaos Engineering for LLM ProductionChaos engineering breaks parts of a system on purpose, under control, to see if it recovers. For AI products that means testing what happens when a model provider is slow, down, or returns bad output.Learn · ~60 minutes
- 25Security — Secrets, API Key Rotation, Audit Logs, GuardrailsThis lesson covers protecting the keys and passwords AI systems use: keeping them in a central vault, rotating them, keeping records of who called what, and adding guardrails on inputs and outputs.Learn · ~60 minutes
- 26Compliance — SOC 2, HIPAA, GDPR, PCI-DSS, EU AI Act, ISO 42001This lesson maps the main compliance rules that touch AI products: SOC 2 for controls, HIPAA for health data, GDPR for personal data, PCI-DSS for card data, plus new AI laws and the ISO 42001 AI management standard.Learn · ~60 minutes
- 27FinOps for LLMs — Unit Economics and Multi-Tenant AttributionFinOps for AI means tracking AI spending per user, per task and per customer, because AI costs come from each request rather than from servers sitting on. Prompt and output length choices become money choices.Learn · ~60 minutes
- 28Self-Hosted Serving Selection — Matching Engine to Hardware and ScaleThis lesson helps choose software for running your own models: llama.cpp for CPUs and devices, Ollama for an easy laptop setup, and vLLM or SGLang for busy GPU servers. The right choice follows the hardware and traffic.Learn · ~45 minutes
- 29Model Demo Interfaces — From a Function to a Shareable AppA model demo is a small web app that takes an input, calls a model and shows the output. Tools like Gradio and Streamlit build one fast. The lesson stresses putting limits on input size, model calls and output before sharing.Build · ~90 minutes
Lessons from AI Engineering from Scratch by Rohit Ghumare and contributors, MIT license. Notes for owners by Lozo. License
Next step
Rather have it built for you?
Reading is optional. Tell me what's stuck and I'll build it into the business you already run.