Production AI, lesson 1
Routing requests between models
Send each request to the cheapest model that can handle it.
A support assistant gets easy questions and hard ones, and a large model can cost twenty times as much per request as a small one. This lesson works out what sending each request to the cheapest model that can handle it saves, and what it risks. You need to know that model calls are priced per token.
By the end, you can:
Part 1. The blended cost of two models
Work out the average cost per request when traffic is split between two models.
A shop's support assistant gets questions like "What time do you close on Sunday?" and "I was charged twice after switching plans, and one charge is in euros. Why?" The first has one answer on a help page. The second needs careful reasoning over billing rules.
Sending both to a large model means paying large-model prices for the easy one too. Here are two real prices, from Anthropic's list in November 2026. Claude Sonnet 5.5, a large model, charges $2 per million input tokens and $10 per million output tokens. Claude Haiku 5.5, a small one, charges $0.10 and $0.50.
Example 1.1
80% of requests to the small model
A typical request reads 2,000 tokens and writes 400. You send 80% of requests to Haiku and 20% to Sonnet. What does a request cost on average, and how does that compare with sending everything to Sonnet?
- Price a request on the large model$0.004 + $0.004 = $0.008
2,000 input tokens at $2 per million cost $0.004. 400 output tokens at $10 per million cost the same.
- Price it on the small model$0.0002 + $0.0002 = $0.0004
The same tokens at a twentieth of the prices.
- Weight each by its share$0.00032 + $0.0016 = $0.00192
80% of $0.0004 is $0.00032, and 20% of $0.008 is $0.0016.
- Compare per million requests$1,920 against $8,000
Multiply both by a million. The routed bill is 1 − 1,920 ÷ 8,000 = 76% smaller.
Answer$0.00192 a request: $1,920 per million requests instead of $8,000, or 76% less.
The bill is an average of the two prices, weighted by how much traffic each model gets. Look at where the money goes. The 20% of requests still on Sonnet cost $1,600 of the $1,920. The 80% on Haiku cost $320.
Blended cost
The average cost per request across your models, each model's cost weighted by its share of traffic. It's the number your monthly bill is made of.
blended = small share × small cost + large share × large cost
The saving is always less than the share you move, because the small model still costs something. Moving 80% saves 0.8 × (1 − 1/20) = 76% here. If the small model cost a third of the large one, the same 80% would save only 0.8 × 2/3, about 53%.
Your turn
Same prices and request size, but now 90% of requests go to Haiku. What does a million requests cost?
Part 2. Quality is a weighted average too
Score the quality a router gives, mistakes included.
A smaller bill is half the story. Say you grade both models on your own questions and get these made-up numbers. On easy questions Haiku is right 96% of the time and Sonnet 98%. On hard ones Haiku drops to 70% and Sonnet manages 93%. Easy questions are 80% of your traffic.
Example 2.1
Three plans, scored
Score three plans: everything on Sonnet, everything on Haiku, and a router that sends easy questions to Haiku and hard ones to Sonnet.
- Score all-large0.8 × 98% + 0.2 × 93% = 97.0%
Weight each kind of question by its share of traffic, as with cost.
- Score all-small0.8 × 96% + 0.2 × 70% = 90.8%
The cheapest plan, but the hard questions drag it down.
- Score the router0.8 × 96% + 0.2 × 93% = 95.4%
Easy questions get Haiku's 96%, and hard ones keep Sonnet's 93%.
Answer97.0%, 90.8% and 95.4%. The router gives up 1.6 points of quality for a 76% saving.
The router keeps most of the quality because it pays the large model's price only where the large model's skill changes the answer. Whether 1.6 points is worth $6,080 per million requests is a product decision. Now it's one you can make with numbers.
Model router
The code that looks at each request and picks the model that answers it. Your app asks for an answer, and the router decides who gives it. Its quality is each model's accuracy on the requests it's sent, weighted by traffic.
quality = each route's share × that model's accuracy, added up
The 95.4% assumes the router never sends a hard question to Haiku by mistake. Each one it does is answered at 70% instead of 93%. A router is a model of its own, and its mistakes are part of your quality.
Your turn
The router sends 1 hard question in 4 to Haiku by mistake. Everything else is as before. What's the quality now?
Part 3. Decide up front, or escalate
Compare the cost and wait of a cascade with deciding up front.
A router that decides up front needs a signal before any answer exists. The simplest are rules: short questions with no attachment go small, anything about refunds goes large. A step up is a classifier, a small model that tags each request easy or hard. Both miss hard requests that look easy.
The other way round is to send every request to the small model first. A check reads its answer: does it parse, does it hedge or refuse, does a quick grader score it low? Only requests whose answer fails go on to the large model.
Example 3.1
The cost and wait of a cascade
Say Haiku answers in 1 s and Sonnet in 3 s, at the prices above. 25% of Haiku's answers fail the check and go on to Sonnet. What does a request cost, and how long does it take, on average?
- Every request pays for the small model$0.0004
Every answer starts on Haiku.
- A quarter also pay for the large one0.25 × $0.008 = $0.002
An escalated request pays for the answer that failed and for the one that replaces it.
- Add them$0.0004 + $0.002 = $0.0024
$2,400 per million requests, against $1,920 for deciding up front.
- Average the wait1 s + 0.25 × 3 s = 1.75 s
Escalated requests wait for both answers in a row, 4 s in all.
Answer$0.0024 a request and 1.75 s on average. The escalated quarter waits 4 s.
The cascade costs more here, and teams use it anyway, because its check judges the actual answer. It catches hard requests that looked easy, which no rule written up front could spot. The escalated requests pay for that twice, in money and in waiting.
Cascade
Cheap model first, escalating to a stronger one only when the first answer fails a check. The share that escalates, the escalation rate, sets both the cost and the wait.
cost = small cost + escalation rate × large cost
Watch the escalation rate after launch. Users drift toward longer, harder questions, and a cascade quietly sends more of them on. A rising rate means the small model is being handed work it can't do, and the bill is climbing back toward all-large.
Your turn
The same cascade, but now 35% of Haiku's answers fail the check. What does a million requests cost?
Where you'll meet it
Asked how you'd cut an AI feature's model bill without hurting it? Start with measurement: grade a sample of real requests on a small and a large model. Then route the ones the small model gets right, or cascade with a check. Give the sum: 80% routed to a model at a twentieth of the price saves 76%.
Research routers find the same trade. RouteLLM (2024) trained routers on human preference data and cut costs by more than half in some tests without lowering quality. FrugalGPT (2023) learned a cascade that matched GPT-4 at up to 98% lower cost on its tasks. Your own graded traffic decides whether that holds for you.
SMALL, LARGE = 0.0004, 0.008 # dollars per request, from part 1
graded = [ # (router's "hard" score, small model right?, large model right?)
(0.05, True, True), (0.10, True, True), (0.15, True, True), (0.20, True, True),
(0.35, True, True), (0.40, False, True), (0.55, True, True), (0.65, False, True),
(0.80, False, False), (0.90, False, True),
]
for cut in (0.0, 0.3, 0.5, 0.7):
big = [g for g in graded if g[0] >= cut] # at or above the cut: large model
right = sum(g[2] if g[0] >= cut else g[1] for g in graded) / len(graded)
cost = (len(big) * LARGE + (len(graded) - len(big)) * SMALL) / len(graded)
print(f"cut {cut}: {len(big)} to large, {right:.0%} right, ${cost * 1e6:,.0f} per million")
# cut 0.0: 10 to large, 90% right, $8,000 per million
# cut 0.3: 6 to large, 90% right, $4,960 per million
# cut 0.5: 4 to large, 80% right, $3,440 per million
# cut 0.7: 2 to large, 70% right, $1,920 per millionThe router scores how hard each request looks, and requests at or above the cut go to the large model. Cut 0.3 keeps all-large quality at 62% of the cost. Cut 0.7 costs part 1's $1,920 but gets two more answers in ten wrong.
Check yourself
Three questions from across the lesson. Try each one before you look back.
1
A request asks the assistant to rewrite "pls send invoice" as a polite one-line email. Which model fits best?
2
A small model costs a third as much per request as a large one. A router sends 80% of requests to the small model. About how much of the all-large bill does it save?
3
A cascade's escalation rate climbs from 20% to 35% over a few months, with no change to your code or models. What's the likeliest cause?
What to remember
1The blended cost of two models
The blended cost weights each model's price by its share of traffic: 80% of requests on a model at a twentieth of the price cost $1,920 per million instead of $8,000, a 76% saving.
2Quality is a weighted average too
Quality is a weighted average too, and every misrouted request gets the wrong model's accuracy: routing scores 95.4% against all-large's 97.0%.
3Decide up front, or escalate
A cascade pays for the small model on every request and the large one on each escalation, $0.0004 + 0.25 × $0.008 = $0.0024 here, and escalated requests wait for both.
Recap and going further
The lesson on one page
- Each model's price, weighted by its share of traffic
- 80% on a model at 1/20 the price saves 76%
- Each model's accuracy, weighted the same way
- A misrouted request gets the wrong model's accuracy
- Rules or a classifier pick before any answer
- They miss hard requests that look easy
- Small model first, large one when a check fails
- Escalations pay twice and wait twice
- Quality per route
- An escalation rate that creeps up
Try this
Grade your own traffic both ways
Take 100 real or realistic requests from your product. Run each through a small and a large model, and grade every answer: by hand for 30, then with a model grader you've checked against your own grades. Write a simple router (keyword rules, or a small model asked how hard each request is) and sweep its cut, as in this lesson's code.
You need: Python and API access to one small and one large model.
Look for
- What share of requests the small model already answers well.
- Which requests it fails, and whether your router can spot them before answering.
- The cheapest cut that keeps quality within a point of all-large.
- How much time the router's own decision adds to each request.
The words, in plain language
- Blended cost
- The average cost per request across your models, each model's cost weighted by its share of traffic. It's the number your monthly bill is made of.
- Model router
- The code that looks at each request and picks the model that answers it. Your app asks for an answer, and the router decides who gives it. Its quality is each model's accuracy on the requests it's sent, weighted by traffic.
- Cascade
- Cheap model first, escalating to a stronger one only when the first answer fails a check. The share that escalates, the escalation rate, sets both the cost and the wait.
Read next
- PricingAnthropic docsInput and output prices per million tokens for each Claude model, the source of this lesson's sums.
- RouteLLM: Learning to Route LLMs with Preference DataIsaac Ong and others, 2024Routers trained on human preference data, and how much they save at a given quality.
- FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving PerformanceLingjiao Chen, Matei Zaharia and James Zou, 2023The cascade idea, with real prices and the savings it found.
- Building A Generative AI PlatformChip Huyen, 2024Where a router sits among the other parts of a production AI system.
Production AI overviewSpotted a mistake? Tell us.