insideout Get Pro

04 · The confusion matrix

Pro

Evaluation, lesson 4

The confusion matrix

What accuracy hides, and precision, recall and F1 per class.

"Read the failures first": sort failures into piles before choosing a number. For a system that sorts things into classes, the first number people reach for is accuracy. This lesson is about what it hides.

Part 1. 95% by never saying yes

A model reads incoming support tickets and flags the urgent ones: outages, security problems, payments stuck. Out of 1,000 tickets, 50 are urgent. Someone ships a version with a bug: it never flags anything.

Your turn

What accuracy does the model that never flags anything get on those 1,000 tickets?

It's right about all 950 tickets that aren't urgent and wrong about the 50 that are: 950 / 1,000 = 95%. A number that high for a model that does nothing is why accuracy is the wrong headline when one class is rare.

Accuracy mixes two very different kinds of right answer. Here, being right about ordinary tickets is easy and plentiful. Being right about urgent ones is the whole point, and accuracy barely notices them.

Part 2. Four boxes

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Four boxes
  2. Precision, recall and the trade between them
  3. One row per class
See Pro plansI have Pro: sign in
Next lessonOverlap metrics: BLEU, ROUGE, BERTScoreWhen matching a reference answer works, and when it misleads. Start lesson 5

Evaluation overviewSpotted a mistake? Tell us.