Evaluation, lesson 11
Is the new version really better
Confidence intervals and paired tests for comparing two systems.
"Specifying a good output" and "Models as graders": scoring a system on a test set. This lesson is about trusting the gap between two scores. You need percentages and square roots, nothing more.
Part 1. Every score has a wobble
Model A gets 52 of 100 test questions right. Model B gets 50. A team member says A is better and wants to switch.
Your turn
If you ran both models on a fresh set of 100 similar questions, how sure are you that A would win again?
The 100 questions are a sample from all the questions users could ask. Draw another sample and the score moves. You can work out how much it moves:
Example 1.1
How wide is 52% on 100 questions?
- p = 0.52n = 100
The score as a share, and the number of questions.
- √(0.52 × 0.48 ÷ 100)=0.05
The typical wobble of a share is √(p × (1 − p) ÷ n). Here that's 0.05, or 5 points.
- 1.96 × 0.05=0.09852% ± 10=42% to 62%
About 95% of fresh samples land within 1.96 wobbles. So the honest report is "somewhere from 42% to 62%". B's 50% sits right in the middle of that.
Confidence interval
A range around a measured score that would contain the true value most of the time (95% of the time, for a 95% interval). Report it next to every score. A score without one hides how little 100 items can tell you.
Part 2. Compare on the same questions
Keep going with Pro
You've done the first part. Pro opens the rest of this lesson, and every other one.
- Compare on the same questions
- Resample and see
- How many questions you need
Evaluation overviewSpotted a mistake? Tell us.