insideout Get Pro

07 · Cross-validation and tuning

Pro

Classical Machine Learning, lesson 7

Cross-validation and tuning

Pick settings on k folds, and keep the test set for the very end.

"Training, validation and test sets" (scores come from held-out rows) and "Regularisation with L1 and L2" (the penalty strength C). The code uses a scaled logistic regression from "Logistic regression".

Part 1. One split is one draw

scikit-learn ships a table of 569 breast tumours, each with 30 measurements and a label: malignant or benign. Hold back 20% as a test set. Split the remaining 455 rows into training and validation, fit a scaled logistic regression, and score it. Do that with five different random splits and the validation accuracy comes out 99.1%, 97.4%, 98.2%, 95.6% and 96.5%.

Nothing changed but which rows landed in validation. With 114 validation rows, each mistake moves the score by almost a point. Compare two settings on one split each and the split, not the setting, can decide the winner.

Your turn

Two settings of a model score 97.4% and 96.5%, each on its own random validation split of 114 rows. What can you conclude?

One mistake in 114 rows is 0.88 points, so the gap is a single row. Above, the very same model ranged from 95.6% to 99.1% on five splits. To compare settings you need a score that doesn't hang on one draw.

The fix is to use every row for validation, once. Cut the 455 rows into five folds of 91. Train on four folds and validate on the fifth, then repeat until each fold has had its turn. You get five scores, and you report their average and their spread.

Example 1.1

Five-fold cross-validation on 455 rows

  1. fold 1fold 2fold 3fold 4fold 5
    90 of 91 right=0.989

    Round 1: train on folds 2 to 5 (364 rows), validate on fold 1.

  2. fold 1fold 2fold 3fold 4fold 5
    88 of 91 right=0.967

    Round 2: a fresh model, trained on folds 1, 3, 4 and 5, validated on fold 2.

  3. round 389 of 91=0.978
    round 488 of 91=0.967
    round 590 of 91=0.989

    Three more rounds, each fold scored by a model that never saw it.

  4. 445 ÷ 455=0.978 ± 0.010

    Every row was validated exactly once. The average is 0.978, and the folds spread around it by 0.010 (their standard deviation).

Cross-validation

Cut the data into k folds; train on k − 1 of them and validate on the one left out, k times over, then average. Every row is validated once. Five and ten folds are the usual choices. Also called k-fold cross-validation.

In practice

Cross-validation costs k fits instead of one. With big data or slow models, people often use one large validation set instead: with hundreds of thousands of rows, a single split barely wobbles.

Part 2. Searching for settings

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Searching for settings
  2. Why the winner's score is too high
  3. Folds that fit the data
See Pro plansI have Pro: sign in
Next lessonDecision treesSplit the rows one question at a time until each group mostly agrees. Start lesson 8

Classical Machine Learning overviewSpotted a mistake? Tell us.