Production AI, lesson 12
Canary, shadow and rollback
Release a change to a few users first, and undo it fast.
"Logs, metrics and traces" and "When to page someone" covered how you see a problem. This lesson is about shipping changes so that a problem reaches as few people as possible.
Part 1. A prompt change is a deploy
Someone adds "Be concise." to the support assistant's system prompt. Answers get shorter. They also stop including the order number, which another system was reading out of the text. Nothing in the code changed, and the refunds team spends the afternoon matching orders by hand.
In an LLM app, the behaviour lives in more than code: the model snapshot, the prompt, retrieval settings, guardrail thresholds, tool descriptions. A change to any of them is a release, and should get the same care: a version number, a review, a test run, and a way back.
Your turn
Your code and prompts didn't change, but answer quality dropped overnight. What's a likely cause?
So a prompt change and a model upgrade go through the same steps: a version number, an eval run against the fixed set, a shadow sample of real traffic, a canary that widens in stages, and a flag that turns it off. Skipping steps is fine for a typo fix. It's a gamble for a new model.
Part 2. Shadow mode: try it on real traffic, show no one
Keep going with Pro
You've done the first part. Pro opens the rest of this lesson, and every other one.
- Shadow mode: try it on real traffic, show no one
- Canary releases and automatic rollback
- Kill switches and A/B tests
Production AI overviewSpotted a mistake? Tell us.