Multimodal AI
For ownersModels that take a photo, a voice note and text together, such as reading a photo of an invoice.
Models that see, hear, read, and reason across modalities.
The lessons
- 01Vision Transformers and the Patch-Token PrimitiveA vision transformer cuts an image into small square patches, turns each patch into a token, and feeds the sequence to the same kind of model used for text. This is how most AI models start to see.Learn · ~120 minutes
- 02CLIP and Contrastive Vision-Language PretrainingCLIP trains an image model and a text model together so a picture and a caption that match end up close in the same number space. It learned this from huge numbers of web images paired with their captions.Build · ~180 minutes
- 03From CLIP to BLIP-2 — Q-Former as Modality BridgeBLIP-2 connects a frozen image model to a frozen language model with a small trained bridge called the Q-Former. The bridge picks out the useful visual details and hands them to the language model as input.Build · ~180 minutes
- 04Flamingo and Gated Cross-Attention for Few-Shot VLMsFlamingo adds special layers to a frozen language model that let it look at images mixed in with text. A learned gate starts closed and opens gradually, so the model gains vision without losing its language skill.Learn · ~120 minutes
- 05LLaVA and Visual Instruction TuningLLaVA links an image encoder to a language model with a simple projection layer, then trains it on conversations about images. The result is a chat assistant that can answer questions about pictures.Build · ~180 minutes
- 06Any-Resolution Vision: Patch-n'-Pack and NaFlexOlder vision models squashed every image into a small square, which blurred fine print and distorted wide or tall shapes. Any-resolution methods keep each image near its real size and shape and pack the patches together efficiently.Build · ~120 minutes
- 07Open-Weight VLM Recipes: What Actually MattersThis lesson sums up what research found matters most when building an open vision-language model: the choice of image encoder, how images connect to the language model, image resolution and, above all, the quality of training data.Learn + lab · ~180 minutes
- 08LLaVA-OneVision: Single-Image, Multi-Image, Video in One ModelLLaVA-OneVision trains one model to handle a single image, several images at once and video, using a staged training plan. Skills learned in one setting carry over to the others.Build · ~180 minutes
- 09Qwen-VL Family and Dynamic-FPS VideoThe Qwen-VL models are an open family of vision-language models. Each version added an idea others copied, such as native image resolution, position encoding that tracks real time in video, and sampling video frames at a variable rate.Learn · ~120 minutes
- 10InternVL3: Native Multimodal PretrainingInternVL3 trains on text and images together from the start of pretraining, instead of building a text model first and attaching vision later. The goal is a model where language and vision skills grow up together.Learn · ~120 minutes
- 11Chameleon and Early-Fusion Token-Only Multimodal ModelsChameleon turns images into discrete tokens from the same vocabulary as text, so one model reads and writes both in a single stream. This is called early fusion because the two are mixed from the very first layer.Build · ~180 minutes
- 12Emu3: Next-Token Prediction for Image and Video GenerationEmu3 shows that one model trained only to predict the next token can generate images and video as well as understand them, once pictures and clips are converted into tokens like text.Learn · ~120 minutes
- 13Transfusion: Autoregressive Text + Diffusion Image in One TransformerTransfusion trains a single transformer with two methods at once: it predicts the next word for text and uses diffusion to create images, keeping images as continuous values instead of converting them to tokens.Build · ~180 minutes
- 14Show-o and Discrete-Diffusion Unified ModelsShow-o uses one transformer that writes text word by word and creates images by filling in masked image tokens over a few rounds. A mixed attention setup lets both jobs share the same model.Learn · ~120 minutes
- 15Janus-Pro: Decoupled Encoders for Unified Multimodal ModelsJanus-Pro uses two separate image encoders inside one model: one tuned for understanding what is in a picture and one tuned for drawing new pictures. Both feed a shared transformer that does the reasoning.Build · ~120 minutes
- 16MIO and Any-to-Any Streaming Multimodal ModelsMIO turns text, images, speech and music into tokens and trains one model on all of them, so it can take any of these as input and produce any of them as output, streaming as it goes.Learn · ~120 minutes
- 17Video-Language Models: Temporal Tokens and GroundingVideo-language models answer questions about video and point to when things happen. Temporal grounding means linking an event or phrase to the exact time span in the clip where it occurs.Build · ~180 minutes
- 18Long-Video Understanding at Million-Token ContextHour-long videos produce far more tokens than any model can read in full. This lesson covers how long-context models and tricks like frame sampling, token pooling and memory let AI work across very long recordings.Build · ~180 minutes
- 19Audio-Language Models: the Whisper to Audio Flamingo 3 ArcThis lesson traces audio AI from Whisper, which turns speech into text, to newer audio-language models that can reason about sound itself, including tone, music, background noise and when events happen in a recording.Build · ~180 minutes
- 20Omni Models: Qwen2.5-Omni and the Thinker-Talker SplitOmni models take in text, images, audio and video and can reply in both text and speech. Qwen2.5-Omni splits the work between a Thinker that reasons and writes, and a Talker that turns that output into natural speech in real time.Build · ~180 minutes
- 21Embodied VLAs: RT-2, OpenVLA, π0, GR00TVision-language-action models take in camera views and a spoken or written instruction, then output movements for a robot. They reuse what vision-language models learned from the web to control physical machines.Learn · ~180 minutes
- 22Document and Diagram UnderstandingDocument understanding is AI that reads PDFs, forms, invoices, tables and diagrams with their layout intact, instead of just pulling out raw text. Newer models read the page image directly without a separate OCR step.Build · ~180 minutes
- 23ColPali and Vision-Native Document RAGColPali searches documents by embedding images of the pages themselves, rather than extracting text first. It matches small parts of the question to small regions of each page, so charts, tables and layout still count.Build · ~180 minutes
- 24Multimodal RAG and Cross-Modal RetrievalMultimodal RAG retrieves across text, images, audio and video, not only documents, and hands the best matches to a model. Cross-modal retrieval means a query in one form, like a photo, can find results in another, like text.Build · ~180 minutes
- 25Multimodal Agents and Computer-Use (Capstone)Computer-use agents look at screenshots, decide where to click or type, and carry out tasks in web pages and apps like a person would. This capstone combines vision, reasoning and tool actions into one loop.Capstone · ~240 minutes
Lessons from AI Engineering from Scratch by Rohit Ghumare and contributors, MIT license. Notes for owners by Lozo. License
Next step
Rather have it built for you?
Reading is optional. Tell me what's stuck and I'll build it into the business you already run.