Production AI, lesson 2
When the main model is down
Fallbacks, timeouts, and degrading gracefully.
Every model API fails some of the time: a call times out, the service is overloaded, or it turns you away. This lesson builds what happens next: retry the blips, fall back to another model, and run with less on purpose. You need to know what an HTTP status code is.
By the end, you can:
Part 1. Calls fail, and some failures are worth retrying
Count how many calls fail in a day, and tell which failures a retry fixes.
At a million model calls a day, even a small failure rate means thousands of people looking at an error. Some failures go away if you ask again a moment later. Others come back every time, because the request itself is the problem.
The status code tells them apart. Anthropic's API, for example, answers 529 when it's overloaded, 500 for an error on its side and 429 when you send too much too fast. A 400 means the request is wrong, such as a prompt too long for the model.
Example 1.1
A day of failed calls
Your app makes 1,000,000 calls a day. Each try fails with a 529, overloaded, 2% of the time, independently of other tries. Another 1,000 calls get a 400 because their prompts are too long. You retry every failure once. How many calls still fail?
- Count the overloaded calls1,000,000 × 2% = 20,000
Bad luck: the same call a moment later will probably get through.
- Retry them once20,000 × 2% = 400
Each retry fails with the same 2% chance, independently.
- Retry the 400s1,000 fail again
The prompt is still too long. The retry only adds load.
- Add up what's left400 + 1,000 = 1,400
Without retries it would be 21,000.
Answer1,400 failed calls a day, most of them 400s that no retry can fix.
One retry turned 20,000 failures into 400 because the two tries failed independently: 2% of 2% is 0.04%. Each extra try multiplies by 2% again. The 400s didn't move, because nothing about them was luck.
Transient error
A failure caused by the provider's state at that moment, not by your request: a timeout, a 429, a 500 or a 529. Retry these. Most other 4xx errors, such as 400 and 401, mean the request or the account needs fixing.
failures after a retry = failures before × chance a try fails
Two cases break that sum. In a real outage every try fails, so retries only add waiting and load. And not every 429 is a blip: in November 2026, Anthropic's 429 for a reached monthly spend cap has no retry-after header, and it keeps failing until access resumes.
Your turn
Your app makes 2,000,000 calls a day. Each try times out 1% of the time, independently. Another 500 calls a day get a 400 for malformed JSON. You retry each timeout up to twice. How many calls still fail?
Part 2. Back off, and add jitter
Work out the average time a request takes when its retries back off with jitter.
A retry sent straight away hits a provider that's still busy. So wait, and double the longest wait each time: up to 1 s before the first retry, up to 2 s before the second. Draw each actual wait at random below that limit.
Example 2.1
The average time with two retries
Each try takes 1 s and fails 20% of the time, independently. You allow two retries: a random wait between 0 and 1 s before the first, between 0 and 2 s before the second. How long does a request take on average?
- Every request makes try 11 s
- 1 in 5 waits and tries again0.2 × (0.5 + 1) = 0.3 s
A wait drawn evenly between 0 and 1 s averages 0.5 s. Then the try takes 1 s.
- 1 in 25 tries a third time0.04 × (1 + 1) = 0.08 s
0.2 × 0.2 = 0.04 get this far, and their wait averages 1 s.
- Add the pieces1 + 0.3 + 0.08 = 1.38 s
Each piece is how often it happens times how long it takes.
Answer1.38 s on average. Only 0.2 × 0.2 × 0.2 = 0.8% of requests fail all three tries.
Retries barely move the average, because most requests never need one. They show up in the slow tail: the 4% that reach a third try take 3 to 6 s. Waiting exactly 1 s, then 2 s, would average 1.52 s.
The randomness, called jitter, is there for the crowd. When a provider stumbles, a thousand clients fail in the same instant, and without jitter they all retry at once.
Exponential backoff with jitter
Doubling the longest wait before each retry (exponential backoff) and drawing the actual wait at random below it (jitter). Drawing it between 0 and the limit is full jitter, and a cap stops the limit growing forever.
retry k waits random(0, base × 2^(k − 1))
Retries also stack, and Anthropic's and OpenAI's SDKs already retry twice by default. Amazon's Builders' Library advises retrying at one point in the stack only.
Your turn
Each try takes 2 s and fails 10% of the time, independently. You allow two retries, with random waits up to 1 s, then up to 2 s. How long does a request take on average?
Part 3. A chain of fallbacks inside a deadline
Fit a chain of fallbacks inside the time a user will wait.
Retries fix blips. When the provider is down for real, you need somewhere else to go: the same model through a second provider or region, then a different model, then an honest "we can't answer right now".
All of it has to fit inside the time a user will wait. So give each request a deadline, and each try a shorter timeout, so one hung try can't use up the whole deadline.
Example 3.1
Two chains, one deadline
The deadline is 10 s and each try times out at 4 s. The primary has stopped answering, and the fallback answers in 2.5 s. Chain A retries the primary once, after a 0.5 s wait. Chain B falls back at once. Does each answer in time?
- Try the primarytimed out at 4 s
Both chains start the same way.
- Chain A retries it4 + 0.5 + 4 = 8.5 s
The provider is still down, so the retry times out too.
- What A leaves the fallback10 − 8.5 = 1.5 s
The fallback needs 2.5 s, so the user gets an error at 10 s.
- Chain B falls back at once4 + 2.5 = 6.5 s
It answers with 3.5 s to spare.
AnswerChain A runs out of time. Chain B answers at 6.5 s.
A retry is a bet that the next try goes better. After a single 529 that's a fair bet. After a timeout it's often a poor one, and losing it costs a whole timeout, so it often pays to fall back straight away.
Fallback chain
The ordered list of places a request goes when the one before it fails, all inside one deadline. Gateways, the proxies teams put in front of their models, let you write it as settings.
time left = deadline − every try and wait so far
In a long outage, chain B still spends 4 s on the dead primary for every request. The fix is to remember: after 3 failures in a row, skip the primary for 30 s, then let one trial call through, and go back to normal if it works. That memory is a circuit breaker.
Your turn
Your deadline is 12 s and each try times out at 5 s. You try the primary twice, with a 1 s wait between, and both time out. How much time is left for the fallback, which needs 3 s?
Part 4. Degrade on purpose
Decide which features stay on a weaker fallback, from its eval scores.
Your prompt was tuned and tested on the primary model. Nobody has measured the fallback on it. Run your eval set through both before any outage, feature by feature. The gaps are rarely even: a smaller model handles some kinds of question nearly as well, and others much worse.
Example 4.1
What stays on the fallback
A support assistant's eval set has 200 order-status questions and 100 billing disputes. The primary passes 196 and 94, the fallback 192 and 61. A feature stays on the fallback only if it's within 5 points of the primary. What runs during an outage?
- Score order status192 ÷ 200 = 96%
The primary scores 196 ÷ 200 = 98%. 2 points apart, so it stays.
- Score billing disputes61 ÷ 100 = 61%
The primary scores 94%. 33 points apart, so it doesn't.
- Count what keeping it would cost100 − 61 = 39 in 100
39 of every 100 billing answers would be wrong, against 6 on the primary.
AnswerOrder status stays on the fallback. Billing disputes go to a person until the primary is back.
The outage now has a plan, written in advance. Easy questions still get answers, and hard ones go to someone who can handle them, not to a model that gets 39 in 100 wrong.
Graceful degradation
Running with fewer features instead of failing outright, decided in advance, feature by feature: a weaker model where its scores hold up, a hand-off to a person where they don't, and a clear message about what's off.
A fallback that only runs in an outage is code nobody has run lately. Amazon's Builders' Library advises running an essential fallback as often as possible in production. So send a small share of everyday traffic down the fallback route, and rerun its eval on every prompt change.
Your turn
On your eval set, the fallback scores 93% on returns questions (the primary 96%) and 58% on warranty claims (the primary 92%). A feature stays on the fallback if it's within 5 points. What runs during an outage?
Where you'll meet it
"Your model provider goes down. What happens to your feature?" Walk one request through: retries with backoff and jitter for transient errors, a deadline, a fallback chain, a circuit breaker so the dead primary isn't tried every time, and degrading feature by feature. Then say how you test the backup path.
Anthropic's SDKs retry connection errors, 429s and 5xx errors twice by default, with backoff, and honour retry-after; max_retries changes that. Gateways such as LiteLLM add fallbacks and cooldowns as settings. A cooldown is their circuit breaker: it takes a failing deployment out of the pool, for 5 seconds by default.
class Breaker:
def __init__(self, max_fails=3, cooldown=30):
self.max_fails, self.cooldown = max_fails, cooldown
self.fails, self.opened = 0, None # opened: when it last opened
def allow(self, now): # closed, or open long enough for one trial
return self.opened is None or now - self.opened >= self.cooldown
def record(self, ok, now):
self.fails = 0 if ok else self.fails + 1
self.opened = now if self.fails >= self.max_fails else None
b = Breaker()
for now, up in [(0, False), (1, False), (2, False), (3, True),
(32, False), (40, True), (62, True)]:
if not b.allow(now):
print(f"t={now}: skip the primary, use the fallback")
continue
b.record(up, now)
print(f"t={now}: primary {'answered' if up else 'failed'}")
# t=0: primary failed
# t=1: primary failed
# t=2: primary failed
# t=3: skip the primary, use the fallback
# t=32: primary failed
# t=40: skip the primary, use the fallback
# t=62: primary answeredThe trial at t=32 fails, so the breaker skips t=40, though the primary would have answered then. It can't know until its next trial, at t=62. That's the price of not hammering a broken provider.
Check yourself
Four questions from across the lesson. Try each one before you look back.
1
Which of these failures should you NOT retry?
2
Your code tries each model call up to 3 times, and the SDK underneath retries each of those tries up to twice. How many calls can one request send to a provider that's down?
3
Your support assistant's primary model is down. The fallback is smaller and weaker at multi-step billing questions. What's the best way to degrade?
4
A circuit breaker opens after 3 failures in a row and lets one trial call through 30 s later. The trial fails. What happens next?
What to remember
1Calls fail, and some failures are worth retrying
Retry only transient errors: with independent 2% failures, one retry turns 20,000 failed calls a day into 400, while a 400 fails every time.
2Back off, and add jitter
Back off with random waits up to 1 s, then up to 2 s: 1 s tries that fail 20% of the time then take 1.38 s on average.
3A chain of fallbacks inside a deadline
The whole chain fits one deadline: retrying a hung primary uses 8.5 s of 10, while falling back at once answers at 6.5 s.
4Degrade on purpose
Degrade feature by feature from eval scores taken before the outage, and run the fallback on everyday traffic so it's tested.
Recap and going further
The lesson on one page
- Timeouts, 429, 500 and 529
- Never a 400: it fails the same way again
- Random waits up to 1 s, then up to 2 s
- Retry at one layer only
- The same model elsewhere, another model, an honest message
- One deadline, a timeout per try
- A circuit breaker skips it for a cooldown
- One trial call decides when to go back
- Keep the features whose fallback scores hold up
- Run the fallback on everyday traffic
Try this
Measure your fallback before you need it
Pick the model you'd fall back to if your main one went down. Run your eval set through it with your prompt unchanged, feature by feature, and write its pass rates next to the main model's. Tune a fallback version of the prompt and run it again. Then put both behind a gateway such as LiteLLM, point the primary at an address that never answers, and time how long requests take to reach the fallback, before and after the cooldown starts.
You need: Python, API keys for two providers, and an eval set of at least 50 cases.
Look for
- Which features hold up on the fallback with the prompt unchanged, and which drop.
- Which failures a tuned prompt fixes, and which remain.
- How long a request waits before the cooldown starts, and after.
- Whether your deadline leaves the fallback enough time to answer.
The words, in plain language
- Transient error
- A failure caused by the provider's state at that moment, not by your request: a timeout, a 429, a 500 or a 529. Retry these. Most other 4xx errors, such as 400 and 401, mean the request or the account needs fixing.
- Exponential backoff with jitter
- Doubling the longest wait before each retry (exponential backoff) and drawing the actual wait at random below it (jitter). Drawing it between 0 and the limit is full jitter, and a cap stops the limit growing forever.
- Fallback chain
- The ordered list of places a request goes when the one before it fails, all inside one deadline. Gateways, the proxies teams put in front of their models, let you write it as settings.
- Graceful degradation
- Running with fewer features instead of failing outright, decided in advance, feature by feature: a weaker model where its scores hold up, a hand-off to a person where they don't, and a clear message about what's off.
Read next
- Timeouts, retries, and backoff with jitterMarc Brooker, Amazon Builders' LibraryHow Amazon picks timeouts, why retries at several layers multiply, and why jitter spreads the load.
- Avoiding fallback in distributed systemsJacob Gabrielson, Amazon Builders' LibraryWhy backup paths that rarely run tend to fail when needed, and how to make them run all the time.
- Claude API errorsAnthropic docsEvery status code the API returns, what each one means, and what the SDKs retry for you.
- Router: load balancing, fallbacks and cooldownsLiteLLM docsA real gateway's settings for retries, fallback chains and skipping a failing deployment.
- Addressing Cascading FailuresGoogle's Site Reliability Engineering bookHow retries and slow fallbacks can turn one failing dependency into a wider outage.
Production AI overviewSpotted a mistake? Tell us.