Why is a scene-level index still needed even when a long-context VLM can read a 2-hour video natively?

Ingesting 100 hours of video into a queryable corpus needs scene-level retrieval, even when individual videos can be read whole

Scene cuts are the only thing a VLM understands

Vector indexes do not support video files

Long-context VLMs cannot stream output

Which three vector types does each scene get in the multi-vector index?

Caption embedding, keyframe embedding, and transcript embedding

Audio waveform, mel-spectrogram, and MFCCs

How are results from the three retrieval streams merged at query time?

Via reciprocal rank fusion across the three ranked lists

By averaging cosine similarities

By picking the caption hit only

By taking the union without scoring

What does the temporal grounding step refine inside the top scene?

The (start, end) timestamp window that contains the answer

The keyframe embedding dimensionality

The transcript word timestamps from Whisper

The number of vectors stored in Qdrant

Which question class is reported separately because it is a known hallucination hotspot?

Counting and action-type questions, where VLMs miscount or mis-order events

Speaker identification questions

Descriptive questions about scenery

Which grounding metric does the rubric measure on a held-out set?

Temporal-grounding intersection-over-union (IoU)

Why does the pipeline require cited timestamps in answers?

They let the viewer jump to the exact (video_id, start, end) so users can verify the claim

Timestamps are required by Whisper output

Timestamps are required by Qdrant payloads

They reduce the embedding dimension

Capstone 12 — Video Understanding Pipeline (Scene, QA, Search)

Twelve Labs productized Marengo + Pegasus. VideoDB shipped the CRUD-for-video API. AI2's Molmo 2 published open VLM checkpoints. Gemini long-context handles hours of video natively. TimeLens-100K defined temporal grounding at scale. The 2026 pipeline is settled: scene segmentation, per-scene caption + embedding, transcript alignment, multi-vector index, and a query that answers with (start, end) timestamps plus frame previews. The capstone is ingesting 100 hours, hitting public benchmarks, and measuring hallucination on counting and action questions.

Lesson 12 of 85Capstone30 hoursPython (pipeline), TypeScript (UI)

Before this: Phase 4 (CV), Phase 6 (speech), Phase 7 (transformers), Phase 11 (LLM engineering), Phase 12 (multimodal), Phase 17 (infrastructure)

On this page

Problem

Long-form video QA is the most bandwidth-hungry multimodal problem at 2026 scale. Gemini 2.5 Pro can read a 2-hour video natively, but ingesting 100 hours of video into a queryable corpus still requires a scene-level index. The production shape combines scene segmentation (TransNetV2 or PySceneDetect), per-scene captioning with a VLM (Gemini 2.5, Qwen3-VL-Max, or Molmo 2), transcript alignment (Whisper-v3-turbo with word timestamps), and a multi-vector index that stores caption, frame embedding, and transcript side by side. The query pipeline answers with (start, end) timestamps plus frame previews.

Benchmarks are public (ActivityNet-QA, NeXT-GQA) plus your own 100-query custom set. Hallucination on counting and action-type questions is the known-hard failure class; the capstone explicitly measures it.

Concept

Three pipelines run in parallel at ingest. Scene segmentation cuts the video into scenes. VLM captioning generates a caption per scene and a frame embedding from a keyframe. ASR alignment produces word-level timestamps. The three streams are joined by (scene_id, time range). Each scene gets three vector types in a multi-vector index (Qdrant): caption embedding, keyframe embedding, transcript embedding.

At query time, the natural-language question fires against all three vectors; results merge with RRF; a temporal-grounding adapter (TimeLens-style) refines the (start, end) window within the top scene. The VLM synthesizer (Gemini 2.5 Pro or Qwen3-VL-Max) takes query + top scenes + cropped frames and answers with cited timestamps and a frame preview.

The hallucination measurement matters. Counting ("how many people enter the room?") and action-type ("does the chef pour before stirring?") questions are notoriously unreliable. Report accuracy separately from descriptive questions.

Architecture

video file / URL
      |
      v
PySceneDetect / TransNetV2  (scene segmentation)
      |
      +--- per-scene keyframe --- VLM caption + frame embedding
      |                            (Gemini 2.5 Pro / Qwen3-VL-Max / Molmo 2)
      |
      +--- audio channel --- Whisper-v3-turbo ASR + word timestamps
      |
      v
multi-vector Qdrant: {caption_emb, keyframe_emb, transcript_emb}
      |
query:
  dense queries against all three -> RRF merge -> top-k scenes
      |
      v
TimeLens / VideoITG temporal grounding (refine start/end within scene)
      |
      v
VLM synth: query + top scenes + frame previews
      |
      v
answer + (start, end) timestamps + frame thumbs + citations

Stack

Scene segmentation: TransNetV2 (state-of-the-art 2024-26) or PySceneDetect
ASR: Whisper-v3-turbo via faster-whisper with word timestamps
VLM captioner + answerer: Gemini 2.5 Pro or Qwen3-VL-Max or Molmo 2
Temporal grounding: TimeLens-100K-trained adapter or VideoITG
Index: Qdrant with multi-vector support (caption / frame / transcript)
UI: Next.js 15 with HTML5 video player and scene thumbnails
Eval: ActivityNet-QA, NeXT-GQA, custom 100-question hand-labeled set
Hallucination benchmark: counting and action-type subsets with hand labels

cf-scene-index

Build It

Ingest walker. Accept YouTube URLs or local MP4s. Downscale to 720p if needed. Persist {video_id, file_path}.
Scene segmentation. Run TransNetV2 or PySceneDetect to produce [{scene_id, start_ms, end_ms, keyframe_path}]. Target 100 hours: ~6k-8k scenes.
ASR pass. Run Whisper-v3-turbo on audio; export word-level timestamps; split into per-scene transcript slices.
VLM captioning. Per scene, call Gemini 2.5 Pro (or Qwen3-VL-Max) with the keyframe and a short caption template. Produce caption + frame embedding.
Multi-vector index. Qdrant collection with three named vectors. Payload: {video_id, scene_id, start_ms, end_ms, keyframe_url}.
Query. Natural-language question fires three dense queries; merge with reciprocal rank fusion; top-k=5 scenes.
Temporal grounding. Run TimeLens-style adapter on the top scene to refine the (start, end) window within the scene.
VLM synth. Call Gemini 2.5 Pro with query + top-3 scene clips (as images or short clips) + transcripts. Require (video_id, start_ms, end_ms) citations.
Eval. Run ActivityNet-QA and NeXT-GQA. Build a 100-query custom set. Report overall accuracy + per-class breakdown (counting, action, descriptive).

Use It

$ video-qa ask --url=https://youtube.com/watch?v=X "how many cars pass the intersection in the first minute?"
[scene]    23 scenes detected
[asr]      transcript complete, 4m12s
[index]    69 vectors written (23 scenes x 3)
[query]    top scene: scene 3 [01:32-01:54], confidence 0.84
[ground]   refined window: [00:12-00:58]
[synth]    gemini 2.5 pro, 1.4s
answer:    5 cars pass the intersection between 00:12 and 00:58.
citations: [scene 3: 00:12-00:58]
          [frame preview at 00:14, 00:27, 00:44, 00:51, 00:57]

Ship It

outputs/skill-video-qa.md is the deliverable. Given a YouTube URL or uploaded video, the pipeline indexes scenes and answers questions with timestamped citations.

Weight	Criterion	How it is measured
25	Temporal grounding IoU	Intersection-over-union on held-out grounding set
20	QA accuracy	NeXT-GQA and custom 100-query
20	Ingest throughput	Hours of video per dollar spent
20	UI and citation UX	Timestamp links, thumbnail strip, jump-to-frame
15	Hallucination rate	Counting and action-type accuracy separately
100

Exercises

Swap Gemini 2.5 Pro for Qwen3-VL-Max on the captioning pass. Report caption quality delta on a human-rated 50-scene sample.
Reduce per-scene frame embedding to one pooled vector instead of multi-vector. Measure the retrieval regression.
Build a "counting strict" mode: the synthesizer extracts each counted instance with a timestamp and the user clicks to verify. Measure whether user-verification reduces hallucination.
Benchmark ingest cost: hours-of-video-per-dollar across three VLM choices. Pick the sweet spot.
Add speaker-diarized transcript: run pyannote speaker diarization on the audio and embed per-speaker transcripts. Demonstrate "what did Alice say about X?" queries.

Key Terms

Term	What people say	What it actually means
Scene segmentation	"Shot detection"	Cutting video into scenes at shot boundaries
Multi-vector index	"Caption + frame + transcript"	Qdrant collection with named vectors per representation
Temporal grounding	"When exactly did it happen"	Refining the (start, end) window for a query answer
Frame embedding	"Visual representation"	A vector embedding of a keyframe; used for scene-visual similarity
RRF fusion	"Reciprocal rank fusion"	Merge strategy across multiple ranked lists; a classic hybrid-retrieval trick
Counting hallucination	"Miscount"	Known failure mode of VLMs on "how many X" questions
ActivityNet-QA	"Video-QA benchmark"	Long-form video QA accuracy benchmark

Quick quiz

Why is a scene-level index still needed even when a long-context VLM can read a 2-hour video natively?
1. Scene cuts are the only thing a VLM understands
2. Vector indexes do not support video files
3. Long-context VLMs cannot stream output
4. Ingesting 100 hours of video into a queryable corpus needs scene-level retrieval, even when individual videos can be read whole
Show the answer
D. Ingesting 100 hours of video into a queryable corpus needs scene-level retrieval, even when individual videos can be read whole
Which three vector types does each scene get in the multi-vector index?
1. Audio waveform, mel-spectrogram, and MFCCs
2. Pose, depth, and optical flow
3. Hash, gzip, and CRC
4. Caption embedding, keyframe embedding, and transcript embedding
Show the answer
D. Caption embedding, keyframe embedding, and transcript embedding
How are results from the three retrieval streams merged at query time?
1. Via reciprocal rank fusion across the three ranked lists
2. By averaging cosine similarities
3. By picking the caption hit only
4. By taking the union without scoring
Show the answer
A. Via reciprocal rank fusion across the three ranked lists
What does the temporal grounding step refine inside the top scene?
1. The (start, end) timestamp window that contains the answer
2. The keyframe embedding dimensionality
3. The transcript word timestamps from Whisper
4. The number of vectors stored in Qdrant
Show the answer
A. The (start, end) timestamp window that contains the answer
Which question class is reported separately because it is a known hallucination hotspot?
1. Translation questions
2. Speaker identification questions
3. Counting and action-type questions, where VLMs miscount or mis-order events
4. Descriptive questions about scenery
Show the answer
C. Counting and action-type questions, where VLMs miscount or mis-order events
Which grounding metric does the rubric measure on a held-out set?
1. Perplexity
2. Temporal-grounding intersection-over-union (IoU)
3. Acceptance rate
4. MRR@10
Show the answer
B. Temporal-grounding intersection-over-union (IoU)
Why does the pipeline require cited timestamps in answers?
1. Timestamps are required by Whisper output
2. Timestamps are required by Qdrant payloads
3. They reduce the embedding dimension
4. They let the viewer jump to the exact (video_id, start, end) so users can verify the claim
Show the answer
D. They let the viewer jump to the exact (video_id, start, end) so users can verify the claim

Questions people ask

Can AI search inside videos?

Yes. The video is cut into scenes, and each scene gets a caption, an embedding and the matching part of the transcript. A search compares the question to those scenes and returns the best matches with timestamps, so you can jump to the right moment.

How does AI find the exact moment in a video that answers a question?

This is called temporal grounding. The system scores scenes or short clips against the question using both visuals and speech, then returns a start and end time for the best match. Good pipelines also show the transcript snippet so a person can check it fast.

Run the code Read it on GitHub

Next lessonCapstone 13: Stateless MCP Server with Registry and Governance Previous: Capstone 11 — LLM Observability & Eval Dashboard

Lessons from AI Engineering from Scratch by Rohit Ghumare and contributors, MIT license. Notes for owners by Lozo. License