insideout Get Pro

06 · Measuring a classifier

Pro

Classical Machine Learning, lesson 6

Measuring a classifier

Read a confusion matrix, and set the threshold by what mistakes cost.

"Logistic regression": a model gives each customer a churn probability, and a threshold turns it into yes or no. "Training, validation and test sets" for why choices are made on validation data.

Part 1. When accuracy lies

Take 1,000 customers, of whom 50 will cancel next month. A rule that says "nobody churns" is right about the 950 who stay and wrong about the 50 who leave. That's 95% accuracy, from a rule that never finds a single churner.

Model M flags 80 customers. 35 of them really do churn; the other 45 would have stayed anyway. It misses the remaining 15 churners. Its accuracy is (35 + 905) ÷ 1,000 = 94%, one point below the rule that does nothing.

Your turn

On 1,000 customers with 50 churners, model M scores 94% accuracy and catches 35 of the 50. The rule "nobody churns" scores 95%. Which is more use to a retention team?

The rule is accurate only because stayers are 95% of customers; it does nothing for the team. Model M trades 45 false alarms for 35 catches. Whether that trade is worth it is a question of cost, but accuracy can't even ask it, because it adds both kinds of mistake into one count.

When one class is rare, accuracy is mostly a measure of the big class. The fix is to keep the four kinds of outcome apart.

Example 1.1

Model M's four boxes

  1. flagged (80)churned: 35stayed: 45

    The 80 customers M flagged. 35 are churners it caught, the true positives. 45 are false alarms, the false positives.

  2. not flagged (920)churned: 15stayed: 905

    The 920 it left alone. 15 are churners it missed, the false negatives. 905 are true negatives.

  3. right: 35 + 905 = 940wrong: 45 + 15 = 60

    Accuracy only ever sees these two totals. The retention team cares a great deal which kind each of the 60 mistakes is.

False positive and false negative

A false positive is a yes that should have been a no: a false alarm. A false negative is a no that should have been a yes: a miss. Their costs are rarely equal.

Confusion matrix

The table of a classifier's four counts: true positives, false positives, false negatives and true negatives. scikit-learn's confusion_matrix puts the real classes in rows and the predictions in columns, so for 0/1 labels it reads [[TN, FP], [FN, TP]].

Part 2. Precision, recall and the trade between them

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Precision, recall and the trade between them
  2. Every threshold at once: ROC and AUC
  3. A threshold set by cost
See Pro plansI have Pro: sign in
Next lessonCross-validation and tuningPick settings on k folds, and keep the test set for the very end. Start lesson 7

Classical Machine Learning overviewSpotted a mistake? Tell us.