insideout Get Pro

09 · Baselines

Pro

Evaluation, lesson 9

Baselines

Compare against the simplest thing that could work.

"The confusion matrix": an accuracy figure depends on how common each class is. This lesson is about what to compare any score against.

Part 1. A score needs a neighbour

A team builds a model that sorts a shop's support tickets into payment, delivery or technical. It gets 82% right on the test set. Everyone's pleased. Then someone counts the test set: 70% of the tickets are about payment.

Your turn

A "model" that labels every ticket as payment, without reading it, would score what?

Seven in ten tickets really are about payment, so always saying payment is right seven times in ten. The model's real contribution is the 12 points above that, not 82. A third (A) is what random guessing among three labels would get.

82% on its own sounded strong. Next to 70% for doing nothing, it's a modest gain. A score only means something next to the score of something simpler.

Baseline

The simplest reasonable system you compare against, run on the same test set with the same metric. The gain you report is the gap between your system and the baseline, not your system's raw score.

Part 2. A ladder of simple things

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. A ladder of simple things
  2. Sometimes the baseline wins
  3. Give the baseline a fair chance
See Pro plansI have Pro: sign in
Next lessonAblationsRemove one part at a time to learn what each part is worth. Start lesson 10

Evaluation overviewSpotted a mistake? Tell us.