Evaluation, lesson 3
Read the failures first
Look at real failures and sort them before choosing a metric.
Ready-made scores measure qualities someone else chose. The failures your users hit are a fact about your own system, and its outputs are the only place to find them. You need pass rates and error bars from "Specifying a good output", and yes or no checks from "Rubrics".
By the end, you can:
Part 1. Notes first, then piles
Turn a sample of outputs into counted piles of failures.
A bike-share company runs a help assistant. It answers questions like "Why was I charged €4?", using live station data and the price list. The log holds 400 conversations and no scores. Someone suggests a library of ready-made scores: helpfulness, coherence, toxicity.
A helpfulness score can rate highly a reply that sends a rider to a dock removed in May. Which failures matter is a fact about this product, so read first: open each conversation beside the data the assistant saw, and write one line on each that went wrong.
Example 1.1
From 100 conversations to five piles
You read 100 conversations and note 38 that went wrong. Grouped, the notes make five piles: stale station data 13, wrong time zone 9, answered one of two questions 7, made-up fee 5, other 4. What share of the failures do the two biggest piles hold?
- Count the failures38 ÷ 100 = 38%
The other 62 went fine.
- Check the piles add up13 + 9 + 7 + 5 + 4 = 38
One note per conversation, so each failure is in one pile.
- Add the two biggest13 + 9 = 22
Stale station data and wrong time zone.
- Find their share22 ÷ 38 = 58%
More than half, and neither pile is about how the model writes.
Answer22 of the 38 failures, 58%.
Notes come before categories. Decide the piles in advance and new problems get filed under old names. Group the notes after reading and the piles fit this system. When a conversation has several problems, note the earliest: an assistant that looked up the wrong station will get the walking time wrong too.
Error analysis and failure mode
Error analysis is reading outputs one by one, noting what went wrong, then grouping and counting the notes. Each pile is a failure mode: one way the system keeps going wrong, such as stale station data.
How many to read? Keep going until new conversations stop adding new kinds of note. After that, more reading mostly sharpens the counts. Do the first round yourself, with whoever knows the product best, because the piles worth having are the ones nobody expected.
Your turn
You read 80 replies from a meeting summariser and note 24 failures: 11 missed action items, 6 wrong names, 4 too long, 3 other. What share of the failures do the two biggest piles hold?
Part 2. Slice the log
Score each slice on its own, and check a gap against its error bars.
Piles say what went wrong. Slices say where. Split the conversations by something you know about each one, such as the rider's language, the kind of question, or new riders against regulars, and score each part on its own. The assistant answers in English and German, so start there.
Example 2.1
English against German
Of the 100 conversations, 75 are in English and 25 in German. 20 English and 18 German conversations failed. What share of each went fine, and is the gap bigger than the error bars?
- Score English(75 − 20) ÷ 75 = 73%
55 of 75 went fine.
- Score German(25 − 18) ÷ 25 = 28%
7 of 25 went fine.
- Error bar on English2 × √(0.73 × 0.27 ÷ 75) = 0.10
73% ± 10, so about 63% to 83%.
- Error bar on German2 × √(0.28 × 0.72 ÷ 25) = 0.18
28% ± 18, so 10% to 46%. Wide, but its top is still under English's bottom.
Answer73% against 28%. The ranges don't overlap, so the German slice really is worse.
In the overall count, German failures hid among three times as many English conversations. The slice also explains the piles: all 5 made-up fees and 6 of the 7 half-answered questions are German. That points at a check: whether the German price list ever reached the prompt.
Slice
A subset of the data picked out by something known about each input, such as a language, a kind of question or a type of customer, and scored on its own. A system can get better overall while one slice gets worse.
Small slices have wide error bars. Had German scored 60%, its range of about 40% to 80% would overlap English's, and 25 conversations couldn't tell you whether German is worse. The way out is to read more German conversations, not to guess.
Your turn
Chats with a support bot go fine on 160 of 200 on the website and 25 of 50 in the phone app. With error bars of 2 × √(p × (1 − p) ÷ n), is the app really worse?
Part 3. From piles to checks
Turn each pile into a fix or a check, and grade the check against your notes.
Now the piles say what to measure. Rank them by how often they happen and how much harm each does; a rare pile that costs money, such as a promised refund, can jump the queue. Then give each pile the cheapest treatment that works.
Made-up fee: code can check that every fee quoted is on the price list. Time zone: fix the cause, converting times in code before the model sees them. Half answered: no code can tell, so a reader, a person or a model, answers one yes or no question: did it answer everything asked?
Example 3.1
Grading the reader
A model reads the same 100 conversations and flags those that left a question unanswered. Your notes say 7 did. The model flags 9, and 6 of those are among your 7. How far can you trust it?
- Count what it caught6 ÷ 7 = 86%
Of the 7 real cases, it flagged 6 and missed 1.
- Count how often a flag is right6 ÷ 9 = 67%
Of its 9 flags, 6 are real and 3 are false alarms.
- Read the mistakes1 miss, 3 false alarms
These 4 conversations show where the yes or no question needs sharper words.
AnswerIt catches 86% of the real cases, and 67% of its flags are right.
The check is graded like a student, against your notes, and you need both numbers. A reader that flags all 100 conversations catches all 7, but only 7 of its flags are right. One that flags nothing raises no false alarms and catches none.
Precision and recall
You met them in "Measuring a classifier". For a check graded against your notes, precision is the share of its flags that are real, and recall is the share of real cases it flags.
precision = 6 ÷ 9 = 67%, recall = 6 ÷ 7 = 86%
Fix causes where you can. Converting times in code is right every time, where a line in the prompt only makes it likely. Then keep the 9 time zone conversations as test cases, so the bug can't come back unnoticed.
Your turn
A code check flags replies that quote a fee missing from the price list. In 200 conversations your notes mark 10 made-up fees. The check flags 8, and all 8 are among your 10. What are its precision and recall?
Where you'll meet it
"Users say the assistant got worse. What do you do?" Don't start with a new metric. Pull a sample of recent conversations, read each beside the data the assistant saw, note what went wrong and group the notes. Slice by something you know, such as language or question type. Then fix or measure the biggest piles.
Hamel Husain describes this on an apartment-leasing assistant. The team built a simple viewer, wrote free-form notes on dozens of conversations, then used a model to group them and counted each group. Three issues covered over 60% of all problems: conversation flow (missing context, awkward replies), failing to hand over to a person, and dates when rescheduling tours.
from collections import Counter
notes = { # failed conversations, one pile each: (language, pile)
("en", "stale data"): 10, ("de", "stale data"): 3,
("en", "time zone"): 7, ("de", "time zone"): 2,
("en", "half answered"): 1, ("de", "half answered"): 6,
("de", "made-up fee"): 5, ("en", "other"): 2, ("de", "other"): 2,
}
piles, langs = Counter(), Counter()
for (lang, pile), k in notes.items():
piles[pile] += k
langs[lang] += k
print(piles.most_common(2), sum(piles.values()))
for lang, n in {"en": 75, "de": 25}.items():
print(lang, f"{(n - langs[lang]) / n:.0%} fine")
# [('stale data', 13), ('time zone', 9)] 38
# en 73% fine
# de 28% fineA spreadsheet does the same job for the first round. Code earns its place when you rerun the same counts after every change, because a new pile usually turns up on top.
Check yourself
Four questions from across the lesson. Try each one before you look back.
1
A team's first evaluation plan is to install a library and track its ready-made scores: helpfulness, coherence, toxicity. What's the main risk?
2
A rubric weights correct 0.4, complete 0.3, clear 0.2 and safe 0.1, with no gates. A reply passes everything but safe. What does it score?
3
Nine failures share one cause: a data feed sends times in UTC, and the assistant passes them on unchanged. What's the best fix?
4
Your piles: 20 replies with awkward wording, and 2 that promised a refund the policy doesn't allow. Which do you work on first?
What to remember
1Notes first, then piles
Read first and let the notes make the piles: 38 failures in 100 conversations, and the two biggest piles held 22 of them, 58%.
2Slice the log
Score each slice on its own and compare error bars: German at 28% ± 18 and English at 73% ± 10 don't overlap, so the gap is real.
3From piles to checks
Give each pile the cheapest fix or check that works, and grade a check against your notes: 6 of 7 caught is 86% recall, 6 right of 9 flags is 67% precision.
Recap and going further
The lesson on one page
- Each output beside everything the system saw
- One line on what went wrong, earliest problem first
- Notes become named piles
- A few piles hold most failures: 22 of 38 here
- Split by language, question type or user group
- Compare error bars before calling a gap real
- Fix causes in code where you can
- Code checks where the answer is exact
- A reader where it isn't, graded on precision and recall
Try this
Read 50 outputs and sort what went wrong
Use logs from a system you work on, or make some: ask any chat assistant 50 real questions from one area (questions from a public forum, or your own work). Put each question and reply in a spreadsheet with a notes column. Read them all and write a line for each one that went wrong. Then group the notes into piles, count them, and add one column you can slice by, such as the kind of question.
You need: A spreadsheet, and 50 outputs from any model or from your own system's logs.
Look for
- How many outputs you read before new ones stopped adding new kinds of failure.
- Whether the biggest pile is about the model's writing or about something around it, like missing or old data.
- Which slice fails most, and whether its error bar clears the others'.
- Which piles you could check with a few lines of code, and which need a reader.
The words, in plain language
- Error analysis and failure mode
- Error analysis is reading outputs one by one, noting what went wrong, then grouping and counting the notes. Each pile is a failure mode: one way the system keeps going wrong, such as stale station data.
- Slice
- A subset of the data picked out by something known about each input, such as a language, a kind of question or a type of customer, and scored on its own. A system can get better overall while one slice gets worse.
- Precision and recall
- You met them in "Measuring a classifier". For a check graded against your notes, precision is the share of its flags that are real, and recall is the share of real cases it flags.
Read next
- A Field Guide to Rapidly Improving AI ProductsHamel HusainWhy looking at your own data beats any tool, and the simple viewers teams build to do it.
- Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human PreferencesShankar and colleagues, 2024Finds that people's criteria change as they grade outputs, which is why reading comes first.
Evaluation overviewSpotted a mistake? Tell us.