Production AI, lesson 13
Cheap decisions before expensive models
A small classifier decides first, and the large model only writes.
"Routing requests between models" chose which model writes each answer, and "Input and output filters" showed a classifier's threshold trading blocks for misses. It helps to have met "Logistic regression" in Classical Machine Learning and "What a model's probabilities are worth" in Transformers. Prices here are examples, not quotes from any provider.
Part 1. Deciding work, and a gate for it
A support inbox gets 10,000 tickets a day. Before anyone answers a ticket, the pipeline makes a few calls about it. Is it spam? Which of five teams should get it? Is it urgent? Does it need a reply at all? Only then, for some tickets, does anything get written.
Your turn
The pipeline does four things with each ticket. Which one really needs a large model?
That's two kinds of work. Writing is open-ended: a reply, a summary, a plan. Deciding picks from a set you know in advance: a route, urgent or not, a person needed or not. To decide, a large model reads the ticket plus your instructions, maybe 2,000 tokens, to return one word out of five. You pay for all that reading, and wait for it, to get a choice a far cheaper tool can often make. ("Routing requests between models" picked which model writes. Here, some steps need no writing at all.)
So put something cheap in front of the large model. It answers the narrow question, and returns a label from the fixed set plus how sure it is. If it's sure enough, plain code acts on the label. If not, the ticket goes on to the large model, or to a person.
Gate
A cheap decision step in front of an expensive one. It answers one question with a label from a fixed set plus a confidence, acts when the confidence is high enough, and passes everything else on. It's a fast path, never the only path.
Three kinds of gate are common, and teams often stack them, cheapest first.
Rules. A sender list for newsletters, a subject that starts "Automatic reply". Free and instant, and blind to anything nobody wrote a rule for.
A small classifier. A logistic regression trained on embeddings of a few hundred tickets you've labelled. It runs on a CPU in well under a millisecond, and scikit-learn's predict_proba gives a probability for every label.
A cheap model with fixed labels. A small chat model held to one of the five labels by structured output. It needs no training data and copes better with new kinds of ticket, but it's the most expensive of the three, and a confidence it writes about itself tends to run high.
Whichever you build, give it one label that means "not mine", such as needs judgement, and never let it act on that label.
Escalation
Passing a case the gate isn't sure about to something stronger: the large model, or a person. A gate that's unsure escalates instead of guessing, so every route needs somewhere to escalate to.
Your turn
Your gate acts on any label at 0.90 or above. A ticket comes back as needs judgement with confidence 0.97. What happens?
Part 2. 10,000 tickets a day, with and without a gate
Keep going with Pro
You've done the first part. Pro opens the rest of this lesson, and every other one.
- 10,000 tickets a day, with and without a gate
- Choosing the threshold from labelled tickets
- Where gates belong, and how they fail
Production AI overviewSpotted a mistake? Tell us.