Evaluation, lesson 1
Specifying a good output
Write down what success looks like before you measure anything.
Before you change a prompt or swap a model, you want a number that says whether the change helped. That number is only as good as the words behind it: what a good answer is, checked how, on which questions. Percentages and one square root are all the maths you need.
By the end, you can:
Part 1. A criterion anyone can check
Turn a vague idea of good into a criterion two people would mark the same way.
A shop's chatbot answers questions about returns. Asked about shoes worn once, it says: "Of course! You have 60 days to return anything." The designer likes the tone. The support lead knows the policy is 30 days, unworn items only. Each is judging by an idea of good that nobody wrote down.
So write one down, as a check against something outside anyone's taste. Here the policy settles it: every policy fact in the reply matches the policy.
Example 1.1
Five replies, one criterion
The policy: returns within 30 days, unworn items only, refund to the original card. Mark five replies pass or fail on that criterion, then count the passes.
- Mark replies 1 and 4pass, pass
"Unworn, within 30 days" and "refund to your original card": every fact matches.
- Mark reply 2fail
"60 days, anything" gets the days and the condition wrong.
- Mark reply 3fail
"Worn once is fine within 30 days": right days, wrong condition. One wrong fact fails a reply.
- Mark reply 5pass
"I can't help with returns" states no policy facts, so none are wrong.
- Count the passes3 ÷ 5 = 60%
Replies 1, 4 and 5.
Answer3 of 5 replies pass, a pass rate of 60%.
Anyone holding the policy reaches the same five marks, because the document settles each one. "Is this a good reply?" can't be settled that way: the designer and the support lead would mark reply 2 differently.
Success criterion
A written statement of one thing a good output must do, worded so that two people checking it give the same mark. A system usually needs a few, one for each way it can fail.
Look again at reply 5. It passes by saying nothing, because a criterion about wrong facts can't fail a reply with no facts in it. One criterion covers one way to fail. Part 3 adds a second, so that saying nothing can't win.
Your turn
A library's policy: books are lent for 21 days and can be renewed twice. Replies: (1) "You can keep a book for 21 days." (2) "You can renew twice, for 28 days each time." (3) "Please ask at the desk." (4) "Renewals are unlimited." How many pass "every policy fact matches the policy"?
Part 2. From a goal to a number
Turn a goal into a metric, a test set and a threshold that pass or fail.
Goals arrive vague: "the bot should be accurate". Nobody can measure that yet. Four answers make it checkable: what accurate means here, what you count on each reply, which questions you count it on, and how high the count must be.
Example 2.1
Accurate, as a number
The team runs the part 1 criterion on 200 real customer questions saved from last month, and 188 replies pass. They agreed beforehand that at least 95% must pass. Does the bot pass?
- Say what accurate meansevery policy fact matches
For a returns bot, accurate means the days, conditions and refunds are right.
- Pick what to countpass or fail per reply
Each reply gets one mark from the criterion.
- Compute the pass rate188 ÷ 200 = 94%
The share of the 200 saved questions whose reply passed.
- Compare with the bar94% < 95%
The bar was set before the run, so the answer is no.
AnswerNo. 94% is under the 95% bar.
Every piece is pinned down now: the criterion, the 200 questions, who marks them and the bar. Next month someone can run the same test on a new version and get a number that means the same thing. Swap in easier questions and the two numbers stop being comparable.
Metric and threshold
A metric is the number you compute, here the share of replies that pass. A threshold is the value it must reach. Both belong to a named test set: the fixed list of inputs you score on.
pass rate = passes ÷ cases = 188 ÷ 200 = 94%
Draw the test set from real use. Questions the team writes come out easier and tidier than what customers type, so 94% on them says little about real traffic. And a metric with no threshold reports a number but decides nothing.
Your turn
A team writes: "Metric: answer quality score. Target: high." What's the biggest problem?
Part 3. Winning the wrong way
Add a guardrail that catches a gain paid for somewhere else.
The bot still gets 12 of the 200 replies wrong. Someone adds a line to the prompt: "Only answer if you are certain." The pass rate jumps, and the change ships. Two weeks later, support tickets are up.
Example 3.1
The fix that refuses
Before the change, the bot answered 196 of the 200 test questions and refused 4, and 12 replies had a wrong fact. After it, the bot answers 160, and 2 replies have a wrong fact. Compare the pass rate and the share answered.
- Pass rate before(200 − 12) ÷ 200 = 94%
The 4 refusals pass, like reply 5 in part 1.
- Pass rate after(200 − 2) ÷ 200 = 99%
All 40 refusals pass too.
- Share answered before196 ÷ 200 = 98%
Nearly everyone got an answer.
- Share answered after160 ÷ 200 = 80%
1 customer in 5 now gets none.
AnswerThe pass rate rose from 94% to 99%, and the share answered fell from 98% to 80%.
The metric did its job: it counted wrong facts, and there were fewer. It couldn't see the cost, which landed on a number nobody was watching. So pair each metric with a check that the gain wasn't paid for somewhere else, and give that check a bar of its own.
Guardrail
A criterion that must hold whatever happens to the main metric, such as "at least 90% of questions get an answer". Failing a guardrail blocks a change on its own. Some allow no failures at all: "never promise a refund the policy doesn't allow".
A guardrail measures what a change could make worse. Ask that question before the run: more refusals, longer replies, slower answers, a bigger bill. Rerunning the main metric can't catch any of those.
Your turn
A coding assistant offers a suggestion on 90 of 100 requests, and 70% of them pass the tests. After a change, 85% pass, but it offers one on only 60 of 100 requests. How many working suggestions per 100 requests, before and after?
Part 4. Decide before you look
Write the decision rule first, with a too-close-to-call band set by the test set's size.
Once results come in, there's a pull to move the bar: 94% feels close enough after a week of work. So write the rule before the run. Ship at 95% passing and 90% answered, block on any invented promise, and say what counts as too close to call.
That depends on the test set. The 200 questions are a sample, and a fresh 200 of the same kind would score a little differently. A rough rule says by how much.
Example 4.1
How far 94% on 200 could move
The bot passes 188 of 200 questions, 94%. Roughly what range could its true pass rate be in?
- Write the score as a sharep = 0.94, n = 200
p is the pass rate, n the number of cases.
- Find one wobble√(0.94 × 0.06 ÷ 200) = 0.017
The usual spread of a pass rate is √(p × (1 − p) ÷ n).
- Double it2 × 0.017 = 0.034
A score lands within two wobbles of the true rate about 19 times in 20.
- Read the range94% ± 3.4 → 91% to 97%
The 95% bar sits inside it.
AnswerAbout 91% to 97%. 200 questions can't tell 94% from 95%: add questions before deciding.
Why a square root? Each case passes or fails partly by luck, and over n cases the luck partly cancels, by the square root of n. So the bar shrinks slowly: four times the cases only halves it. 800 questions would still leave 94% ± 1.7.
Error bar
The range a score could move if you reran it on fresh cases of the same kind: about two wobbles either way. "Is the new version really better" turns it into a proper confidence interval.
error bar ≈ 2 × √(p × (1 − p) ÷ n)
Small test sets hide small changes. Near 80% on 100 cases the bar is ± 8 points, so 83% hasn't shown it's better. Even on the same 100 cases, a 3-point gain can be just 3 cases that changed, and a version that's no better wins 3 coin flips in a row 1 time in 8.
Your turn
A summariser passes 90 of 100 test cases. Using the error bar, could its true pass rate be 95%?
Where you'll meet it
"How would you know a new prompt is better?" is a common question. Answer in order. Say what good means, as checks two people would mark the same way. Name the test set, drawn from real traffic, and the bar. Add a guardrail for what the change might cost. Write the decision rule first, with an error bar: about ± 8 points at 80% on 100 cases.
Anthropic's guide to success criteria makes the same move. "The model should classify sentiments well" becomes an F1 score of at least 0.85 on a held-out set of 10,000 varied Twitter posts: a metric, a bar and a named test set. Its safety example has a bar too: under 0.1% of 10,000 outputs flagged as toxic.
from math import sqrt
def decide(passes, answered, n, bar=0.95, answer_bar=0.90):
p = passes / n
e = 2 * sqrt(p * (1 - p) / n)
tag = f"{p:.0%} ± {e:.1%}: "
if answered / n < answer_bar:
return tag + "hold, guardrail failed"
if p - e >= bar:
return tag + "ship"
return tag + ("hold" if p + e < bar else "too close, add cases")
print(decide(188, 196, 200))
print(decide(198, 160, 200))
print(decide(1960, 1950, 2000))
# 94% ± 3.4%: too close, add cases
# 99% ± 1.4%: hold, guardrail failed
# 98% ± 0.6%: shipThe rule is ordinary code, so keep it in a file beside the prompt and change it on purpose, with a reason. "Treating prompts like code", later in this module, runs checks like these on every change.
Check yourself
Three questions from across the lesson. Try each one before you look back.
1
You want to check a returns bot on 200 real questions. Which instruction to the checker gives you a number you can trust?
2
Version B passes 83 of 100 test cases and version A passes 80. What can you conclude?
3
A rule written before the run says: ship if at least 95% of replies have every fact right and at least 90% of questions get an answer. The run shows 97% and 86%. What should the team do?
What to remember
1A criterion anyone can check
A success criterion is one check two people would mark the same way, such as "every policy fact matches the policy": 3 of the 5 replies passed it.
2From a goal to a number
A metric needs a named test set and a threshold: 188 of 200 real questions is 94%, which fails a 95% bar.
3Winning the wrong way
Pair the main metric with a guardrail: refusing more lifted the pass rate from 94% to 99% but cut the share answered from 98% to 80%.
4Decide before you look
Write the decision rule before the run, with an error bar of about 2 × √(p × (1 − p) ÷ n): 94% on 200 cases means about 91% to 97%.
Recap and going further
The lesson on one page
- One checkable statement each
- Worded so two people give the same mark
- One criterion covers one way to fail
- A pass rate on a named test set
- Test cases drawn from real use
- A bar that turns the number into yes or no
- Must hold whatever else improves
- Catch a gain paid for elsewhere, like more refusals
- Ship, hold, or too close to call
- Written before any results come in
- Error bar ≈ 2 × √(p × (1 − p) ÷ n)
Try this
Write the criteria for an assistant you use
Pick an assistant you use: the support chat on a shop's site, a coding assistant, a meeting summariser. Write three success criteria, each with a threshold, and one guardrail. Collect 20 real questions, run them, and mark every reply in a spreadsheet, one column per criterion. Then ask a friend to mark the same 20 replies without talking to you.
You need: A spreadsheet and any chat assistant. No code.
Look for
- Which criterion you and your friend marked differently most often, and which words in it caused that.
- Any reply that was bad in a way none of your criteria caught.
- The error bar on each pass rate: on 20 questions at 70%, it's about ± 20 points.
- Whether the guardrail ever failed, and what those replies had in common.
The words, in plain language
- Success criterion
- A written statement of one thing a good output must do, worded so that two people checking it give the same mark. A system usually needs a few, one for each way it can fail.
- Metric and threshold
- A metric is the number you compute, here the share of replies that pass. A threshold is the value it must reach. Both belong to a named test set: the fixed list of inputs you score on.
- Guardrail
- A criterion that must hold whatever happens to the main metric, such as "at least 90% of questions get an answer". Failing a guardrail blocks a change on its own. Some allow no failures at all: "never promise a refund the policy doesn't allow".
- Error bar
- The range a score could move if you reran it on fresh cases of the same kind: about two wobbles either way. "Is the new version really better" turns it into a proper confidence interval.
Read next
- Define success criteria and build evaluationsAnthropic, Claude docsTurning a vague goal into specific, measurable criteria, with example metrics for common tasks.
- Your AI Product Needs EvalsHamel HusainA real estate assistant taken from vibe checks to a working set of tests, step by step.
- Adding Error Bars to Evals: A Statistical Approach to Language Model EvaluationsEvan Miller, Anthropic, 2024Standard errors and 95% intervals for eval scores, and why to compare two models question by question.
Evaluation overviewSpotted a mistake? Tell us.