What is the core idea of model cascading?

Run a cheap model first, escalate to a frontier model only on low confidence or refusal

Run two models in parallel and average the outputs

Run every request on the most expensive model first

Which four signals does the lesson list for routing decisions?

Task classification, prompt length, embedding similarity to known-hard set, and self-confidence from a first-pass

GPU temperature, fan speed, room humidity, time of day

What is the expected latency profile of a cascade router?

About 1.2x median latency (cheap run plus verify), about 2x on escalated requests (~10% of traffic)

Identical to the frontier model

What is cheap-model drift in routing?

Task distribution shifts but the trained router keeps sending requests to the cheap model, silently degrading quality

The cheap model becomes more expensive

Cascade falls through to frontier 100% of the time

A latency drift in the cheap model

Which guard does the lesson recommend to catch routing drift?

Online quality metrics — thumbs-up/down per route, LLM-judge on held-out samples, escalation rate, refusal rate

Quarterly engineering review only

Model Routing as a Cost-Reduction Primitive

A dynamic broker evaluates every request (task type, token length, embedding similarity, confidence) and sends simple queries to a cheap model, escalating complex ones to a frontier model. Also called model cascading. Production case studies show 20-60% cost reduction at iso-quality across US/UK/EU deployments; a 30% routing efficiency improvement on high-volume SaaS turns into six-figure annual savings. The 2026 context is that LLM inference prices for a fixed capability level have dropped about 10x per year (a16z LLMflation estimate); at the GPT-4 tier the drop is slower, from $20/M to ~$0.40/M (about 50x) from late 2022 to 2026. Most of the drop is better serving stacks (Phase 17 · 04-09), not hardware. Routing is how you convert that price drop into margin without product regression. The failure mode is cheap-model drift: the route pushes 40% to a weaker model, quality drops 3-5% on reasoning tasks, no one notices for a quarter. Gate routes by online quality metrics, not just offline eval sets.

Lesson 16 of 29Learn~60 minutesPython (stdlib, toy cascading router simulator)

Before this: Lesson 1 of Phase 17 (Managed LLM Platforms), lesson 19 of Phase 17 (AI Gateways)

On this page

Learning Objectives

Explain model cascading: cheap-first with confidence check, escalate on low confidence.
Enumerate the four routing signals (task classification, prompt length, embedding similarity to known-hard set, self-confidence from first-pass).
Compute expected blended cost at target routing split and quality loss tolerance.
Name the drift-monitoring metric (online quality gate) that catches cheap-model creep.

The Problem

Your service costs $80k/month on GPT-5. Your analytics show 70% of queries are simple: "what time is it in Paris?" "rephrase this sentence." A Haiku-class model handles those perfectly at 3% of the cost. 30% need GPT-5's reasoning — coding, math, multi-step planning.

If you route the 70% to cheap and 30% to expensive, your bill drops ~65% at the same product quality. This is routing. The trick is building the broker without regressing quality.

The Concept

Four routing signals

Task classification: simple/complex/codegen/math/chat. Can be a rules-based classifier, a small LLM (Haiku-class at $0.25/M), or embedding similarity to labeled buckets. Output: route = cheap / balanced / frontier.
Prompt length: prompts >4K tokens often need frontier for coherence. Prompts <500 tokens usually don't.
Embedding similarity to known-hard set: if the query is close (cosine > 0.88) to a known-hard bucket, escalate to frontier directly.
Self-confidence from first-pass: send to cheap; if model's log-probs show low confidence OR it refuses OR outputs hedging language, retry on frontier. Adds P95 latency on ~10% of traffic but saves 50%+ on the other 90%.

Three patterns

Pre-route (classifier up front): ~5-10ms latency added; fastest overall.

Cascade (cheap-first, escalate on low confidence): ~1.2x median latency (cheap run plus verify), ~2x on escalated. Best quality floor.

Ensemble route (run cheap and frontier in parallel for a sample, reward-model pick): highest quality, highest cost; use only for critical A/B.

Implementation

AI gateways (Phase 17 · 19) expose routing. LiteLLM has router config with fallback and cost-routing. Portkey has guards + routing. Kong AI Gateway has plugin-based routing. OpenRouter's model marketplace exposes a recommendation API.

Open-source: RouteLLM (LMSYS), Prompt Mule. Commercial: Not Diamond.

The 2026 price curve

Model class	Late 2022	2026	Change
GPT-4-level quality	~$20/M	~$0.40/M	50x cheaper
Frontier (GPT-5, Claude 4)	—	~$3-10/M	new tier

Most of the improvement is serving efficiency — the core lessons in Phase 17 · 04-09 turned into provider-side cost drops. Routing lets you capture those gains at the app layer instead of waiting for all your users to migrate to the cheap tier.

Drift is the real risk

Your route sends 40% to the cheap model. Over six months, the task distribution shifts (users get more sophisticated, ask longer questions). The router doesn't notice because its classifier was trained on Q1 data. Quality drops silently. Nobody complains loud enough. You find out in a competitor benchmark you lost.

Gate routes by online quality metrics:

User thumbs-up / thumbs-down per route.
Automated LLM-judge on a held-out sample (5%) per route.
Escalation rate: if cascade is kicking up-route >30%, the cheap model is being over-routed.
Refusal rate per route.

Numbers you should remember

2026 routing savings at iso-quality: 20-60% case studies.
LLM price drop at a fixed capability level: ~10x per year (a16z LLMflation estimate); about 50x total at the GPT-4 tier from 2022 to 2026.
GPT-4-level 2022 vs 2026: ~$20/M → ~$0.40/M.
Cascade latency impact: ~~1.2x median, ~2x escalated (~~10% of traffic).

model-cascade-router

Use It

code/main.py simulates pre-route, cascade, and ensemble on a mixed workload. Reports blended cost, quality loss, and escalation rate.

Ship It

This lesson produces outputs/skill-router-plan.md. Given workload and quality budget, picks a routing pattern and signals.

Exercises

Run code/main.py. At what accuracy floor does cascade beat pre-route?
Your user base is 30% enterprise (complex queries), 70% free tier (simple). Design the routing split. What online metric gates it?
A route drops quality by 2% but saves 40%. Is that a ship? Depends on product — argue both.
Implement a confidence check using logprobs from OpenAI / Anthropic APIs. What's the threshold you start with?
Over six months, escalation rate climbs from 8% to 22%. Diagnose three causes and the fix for each.

Key Terms

Term	What people say	What it actually means
Model routing	"cost broker"	Dynamic choice of model per request
Model cascade	"cheap-first escalate"	Run cheap, fall through to frontier on low confidence
Pre-route	"classify first"	Classifier up front; no re-run
Ensemble route	"parallel pick"	Run multiple, reward-model picks best
Escalation rate	"uprouted %"	Fraction of cascade requests that escalated
RouteLLM	"LMSYS router"	OSS router library
Not Diamond	"commercial router"	SaaS model-routing product
Drift	"cheap creep"	Distribution shift without router noticing
Online quality gate	"live check"	Automated LLM-judge sampling live traffic

Quick quiz

What is the core idea of model cascading?
1. Always route by random weight
2. Run two models in parallel and average the outputs
3. Run every request on the most expensive model first
4. Run a cheap model first, escalate to a frontier model only on low confidence or refusal
Show the answer
D. Run a cheap model first, escalate to a frontier model only on low confidence or refusal
Which four signals does the lesson list for routing decisions?
1. User tier only
2. Token count only
3. Task classification, prompt length, embedding similarity to known-hard set, and self-confidence from a first-pass
4. GPU temperature, fan speed, room humidity, time of day
Show the answer
C. Task classification, prompt length, embedding similarity to known-hard set, and self-confidence from a first-pass
What is the expected latency profile of a cascade router?
1. Always faster than pre-route
2. About 1.2x median latency (cheap run plus verify), about 2x on escalated requests (~10% of traffic)
3. Always slower by 10x
4. Identical to the frontier model
Show the answer
B. About 1.2x median latency (cheap run plus verify), about 2x on escalated requests (~10% of traffic)
Which routing pattern adds 5-10ms latency up front and is fastest overall?
1. Cascade
2. Ensemble route
3. Random round-robin
4. Pre-route with a classifier
Show the answer
D. Pre-route with a classifier
What is cheap-model drift in routing?
1. The cheap model becomes more expensive
2. Cascade falls through to frontier 100% of the time
3. A latency drift in the cheap model
4. Task distribution shifts but the trained router keeps sending requests to the cheap model, silently degrading quality
Show the answer
D. Task distribution shifts but the trained router keeps sending requests to the cheap model, silently degrading quality
Which guard does the lesson recommend to catch routing drift?
1. Online quality metrics — thumbs-up/down per route, LLM-judge on held-out samples, escalation rate, refusal rate
2. Quarterly engineering review only
3. Only offline eval sets
4. Disable routing in production
Show the answer
A. Online quality metrics — thumbs-up/down per route, LLM-judge on held-out samples, escalation rate, refusal rate

Questions people ask

What is an LLM router?

An LLM router is a layer that looks at each incoming request and decides which model should answer it. It may use rules, request length, a small classifier, or a confidence check. Simple requests go to cheaper models and harder ones go to stronger models.

Does routing to cheaper models lower answer quality?

It can if the router misjudges hard questions as easy. Good setups measure quality on a test set, send uncertain cases to the stronger model, and watch results over time. Done well, most users see the same quality at lower cost.

Run the code Read it on GitHub

Next lessonDisaggregated Prefill/Decode — NVIDIA Dynamo and llm-d Previous: Batch APIs — the 50% Discount as Industry Standard

Lessons from AI Engineering from Scratch by Rohit Ghumare and contributors, MIT license. Notes for owners by Lozo. License