Classical Machine Learning, lesson 3
Training, validation and test sets
Keep rows aside, so your score comes from data the model never saw.
A model can score perfectly on the rows it was fitted on and still guess badly on new ones. This lesson shows why, and how to set rows aside so the score you report is honest. You need mean squared error and fitting from "Linear regression".
By the end, you can:
Part 1. Overfitting and underfitting
Spot overfitting and underfitting from training error and new-data error.
Here are 12 measurements of something that follows a smooth wave, taken at even steps from x = 0 to x = 1. Each carries noise, typically about 0.25 above or below the wave. You don't know the wave: you only see the 12 points. Fit curves of rising flexibility to them, and score each one twice.
Example 1.1
Four curves, two scores
Fit polynomials of degree 1, 3, 7 and 11 to the 12 points. Score each by its MSE on those 12, and on 30 new points from the same wave. Which degree predicts new points best?
- Degree 1, a straight linetraining 0.197, new 0.269
A line can't follow a wave. It misses old and new points alike.
- Degree 3, two bendstraining 0.021, new 0.061
Enough for the wave. 0.061 is about the noise itself, 0.25² = 0.0625, which no curve can reliably beat.
- Degree 7training 0.013, new 0.079
Extra bends shave the training error by following the noise.
- Degree 11, 12 numberstraining 0.000, new 3.701
A number per point, so it hits all 12, and swings far off the wave between them.
AnswerDegree 3. Degree 11 is perfect on the 12 points and about 60 times worse on new ones.
Twelve noisy points don't pin down a wave, only twelve dots. A curve with a number for every dot can pass through all of them, and to do it, it treats the noise as part of the pattern. The noise is different on new points, so the curve is wrong there.
Overfitting and underfitting
Overfitting is fitting the noise in the training rows along with the pattern: training error keeps falling while error on new data rises. Underfitting is a model too simple for the pattern: error is high on training and new data alike.
So a training error of 0 is a warning sign, not a success. And no model beats the noise: degree 3's 0.061 is about the floor here. If a model's error on new data comes out far below what the noise allows, look for a bug or a leak before you trust it.
Your turn
Model P scores an MSE of 0.02 on its training rows and 0.90 on new rows. Model Q scores 0.30 and 0.32. The noise alone gives about 0.06. Which is which?
Part 2. Three piles of data
Give training, validation and test rows one job each.
The fix is a habit: before you fit anything, set rows aside where the model can't see them, in three piles. A common split is 60/20/20. With millions of rows, 98/1/1 is plenty, since 1% of a million is still 10,000.
Example 2.1
Choosing a degree honestly
You have the 12 training points, plus 30 validation points and 30 test points from the same wave. Choose a degree from 0, 1, 3, 5, 7, 9 and 11, and say how well it will do on new data.
- Fit every candidate on the training points7 curves
The validation and test points play no part in fitting.
- Score each on the validation pointsdegree 3: 0.061, degree 5: 0.062
Degree 3 has the lowest error, and degree 5 is a hair behind.
- Score the winner once on the test pointsdegree 3: 0.069
That's the number to report, close to the noise level of 0.0625.
AnswerDegree 3, with a test MSE of 0.069 to report.
Why two held-out piles and not one? Because choosing is a kind of fitting. Try 40 settings and keep whichever scores best on some rows, and those rows have helped pick the winner. Part of its score is luck on exactly those rows, so you need fresh rows to measure it.
Training set, validation set and test set
The training set is the rows a model is fitted on. The validation set is rows kept out of fitting and used to choose: which model, which degree, which settings. The test set is kept out of everything until the end, then scored once to estimate how the choice will do on new data.
With few rows, one validation pile is a noisy judge: redraw the 30 points 1,000 times and degree 3 wins only 344 of them, degree 5 wins 442. "Cross-validation and tuning" deals with that. And once you've looked at the test score it's spent: change the model because of it, and you need fresh rows.
Your turn
A team tries 40 settings and keeps the one with the best score on the test set: 91%. What's the honest reading of that 91%?
Part 3. Bias and variance
Tell bias from variance, and pick the fix that matches.
A model can be wrong in two different ways, and they need opposite fixes. To see them, collect the 12 measurements again and again (same x positions, fresh noise each time), refit each curve every time, and look at what each one predicts at a single spot.
Example 3.1
2,000 fresh draws, one prediction each
At x = 0.25 the true wave is exactly 1.0. Redraw the 12 points 2,000 times and refit degrees 1, 3 and 11 each time. Where do their predictions at x = 0.25 land? (The spread is the standard deviation of the 2,000 predictions.)
- Degree 1average 0.36, spread 0.09
Far below 1.0 every time: wrong the same way.
- Degree 3average 0.96, spread 0.13
Close to 1.0, and one fit is usually within about 0.13 of that.
- Degree 11average 1.00, spread 0.33
Right on average, but one fit can be far off. Near the edge, at x = 0.95, its spread is 3.5.
- Degree 11, with 100 pointsspread at x = 0.95: 0.10
More data calms the swings.
AnswerDegree 1 is wrong the same way every time, degree 11 jumps from draw to draw, and degree 3 keeps both small.
For squared error, the expected miss on a new point splits into three parts: the bias squared, the variance, and the noise in the label, which no model can remove. A more flexible model usually trades bias for variance. More data lowers variance and leaves bias alone.
Bias and variance
Bias is error that stays however often you redraw the data: the model's average answer is off, because it's too simple for the pattern. Variance is how much a model's answers change from one sample of rows to another.
expected squared error = bias² + variance + noise
To tell which you have, draw a learning curve: training and validation error as the training pile grows. Two lines far apart and still closing mean variance, and more data will help. Two lines that have met at a high error mean bias, and more rows won't move them.
Your turn
A model's training error and validation error are both high, and close to each other. Which change is most likely to help?
Part 4. Splits that match real use
Split by customer and by time, so the test looks like real use.
A random split treats every row as a fresh, unrelated case. Often they aren't. A churn table might hold one row per customer per month, and one customer's months look alike. Split those rows at random and the test pile fills with customers the model has already met.
Example 4.1
One customer, twelve rows
A churn table has 1,000 customers with 12 monthly rows each. Each row goes to the test pile with chance 0.2, at random. How many customers end up with rows on both sides?
- Chance a customer has no test rows0.8¹² ≈ 0.069
Each of their 12 rows stays in training with chance 0.8.
- Chance of rows on both sides1 − 0.069 − 0.2¹² ≈ 0.931
0.2¹², the chance all 12 rows go to test, is practically 0.
- Scale to 1,000 customers1,000 × 0.931 ≈ 931
Nearly every test customer also sits in training.
AnswerAbout 931 of the 1,000. Almost no test customer is new to the model.
The model can score by recognising customers it has already seen, which it can't do for new ones. Split by customer instead, all of each one's rows on one side, and the score drops to what you'll really get. scikit-learn's GroupShuffleSplit splits by a customer column.
Group split and time split
A group split keeps all the rows of one customer, patient or house on the same side. A time split trains on older rows and tests on later ones, the way the model will be used: on a future it hasn't seen.
If the model will predict next month, test it that way: a random split lets it learn from rows dated after the ones it's tested on. And fix the seed, with random_state=0, so a change in score comes from your change, not a reshuffle.
Rare labels need care too. With 20 cancellations in 1,000 rows, a random 200-row test pile should hold 4, but about one pile in ten holds 1 or fewer, or 8 or more. stratify=y keeps exactly 4 in it.
Your turn
A churn table has one row per customer per month, a year of rows for each customer. You split the rows at random into training and test. What goes wrong?
Where you'll meet it
A common question: your model scored well offline and badly once it went live. Why? Name the usual causes: settings chosen on the test set, one customer's rows on both sides, a random split where time mattered, or a column that wasn't really known at prediction time.
In scikit-learn, train_test_split shuffles the rows and holds back 25% for testing unless you say otherwise. For data in time order, TimeSeriesSplit makes splits where every test fold comes after its training rows, so the model never trains on the future.
import numpy as np
from numpy.polynomial import Polynomial
rng = np.random.default_rng(39)
wave = lambda x: np.sin(2 * np.pi * x) # the true pattern, unknown in real life
x_train = np.linspace(0, 1, 12)
y_train = wave(x_train) + rng.normal(0, 0.25, 12)
x_val = rng.uniform(0, 1, 30)
y_val = wave(x_val) + rng.normal(0, 0.25, 30)
x_test = rng.uniform(0, 1, 30)
y_test = wave(x_test) + rng.normal(0, 0.25, 30)
def mse(curve, x, y):
return np.mean((y - curve(x)) ** 2)
curves = {}
for degree in [0, 1, 3, 5, 7, 9, 11]:
curves[degree] = Polynomial.fit(x_train, y_train, degree) # fitted on the 12 only
print(f"degree {degree:>2}: training {mse(curves[degree], x_train, y_train):.3f}, "
f"validation {mse(curves[degree], x_val, y_val):.3f}")
best = min(curves, key=lambda d: mse(curves[d], x_val, y_val)) # chosen on validation
print(f"picked degree {best}; test error, measured once: {mse(curves[best], x_test, y_test):.3f}")
# degree 0: training 0.379, validation 0.594
# degree 1: training 0.197, validation 0.269
# degree 3: training 0.021, validation 0.061
# degree 5: training 0.021, validation 0.062
# degree 7: training 0.013, validation 0.079
# degree 9: training 0.011, validation 0.080
# degree 11: training 0.000, validation 3.701
# picked degree 3; test error, measured once: 0.069This is part 2 in code: every curve fitted on the 12 training points, the degree chosen on validation, and one test score at the end. The training column never rises as the degree goes up, which is why it can't be the judge.
Check yourself
Three questions from across the lesson. Try each one before you look back.
1
A model's training error is 0.01 and its validation error is 0.40. Which change is most likely to help?
2
The labels carry noise with a standard deviation of 0.5. A model's MSE on new rows comes out at 0.05. What's the most likely explanation?
3
A model will forecast each week's sales from two years of daily rows. How should you split them?
What to remember
1Overfitting and underfitting
Training error alone misleads: degree 11 scored 0.000 on its 12 points and 3.701 on new ones, while degree 3 scored 0.061, about the noise level.
2Three piles of data
Fit on training rows, choose on validation rows, then score the winner once on test rows: degree 3's 0.069 is the number to report.
3Bias and variance
Both errors high and close means bias, so try a more flexible model or better features; a wide gap means variance, so add rows or simplify.
4Splits that match real use
Split the way the model will be used: a random row split left about 931 of 1,000 customers on both sides, so split by customer, and by time when you forecast.
Recap and going further
The lesson on one page
- Training error near 0, new-data error high
- The model learned the noise along with the pattern
- Both errors high and close together
- The model is too simple for the pattern
- Training: fit the numbers
- Validation: choose between models
- Test: measure once, at the end
- Bias: wrong the same way every time
- Variance: changes with whichever rows you got
- More data cuts variance, not bias
- One customer's rows on one side
- Past to train, future to test
- A fixed seed; stratify rare labels
Try this
Draw learning and validation curves on real data
Load scikit-learn's diabetes data (load_diabetes, 442 patients with 10 measurements each) and split off a 25% test pile with random_state=0. On the rest, draw learning curves with LearningCurveDisplay.from_estimator for two models: LinearRegression() and DecisionTreeRegressor(random_state=0) with no depth limit. Then draw a validation curve for the tree's max_depth from 1 to 15 with ValidationCurveDisplay.from_estimator. Only at the end, score your chosen depth once on the test pile.
You need: Python with scikit-learn and matplotlib, or a free notebook such as Google Colab.
Look for
- The unlimited tree's training score stuck at a perfect 1.0, with a wide gap below it.
- The linear model's two curves meeting early: little variance, and little help from more rows.
- The depth where the validation score peaks, and how flat the curve is around it.
- How the test score compares with the best validation score.
The words, in plain language
- Overfitting and underfitting
- Overfitting is fitting the noise in the training rows along with the pattern: training error keeps falling while error on new data rises. Underfitting is a model too simple for the pattern: error is high on training and new data alike.
- Training set, validation set and test set
- The training set is the rows a model is fitted on. The validation set is rows kept out of fitting and used to choose: which model, which degree, which settings. The test set is kept out of everything until the end, then scored once to estimate how the choice will do on new data.
- Bias and variance
- Bias is error that stays however often you redraw the data: the model's average answer is off, because it's too simple for the pattern. Variance is how much a model's answers change from one sample of rows to another.
- Group split and time split
- A group split keeps all the rows of one customer, patient or house on the same side. A time split trains on older rows and tests on later ones, the way the model will be used: on a future it hasn't seen.
Read next
- Datasets: Dividing the original datasetGoogle for DevelopersWhy a validation set sits between the training and test sets, and what wears a test set out.
- Underfitting vs. Overfittingscikit-learn examplesPolynomials of degree 1, 4 and 15 fitted to a noisy curve, each scored on held-out data.
- Validation curves: plotting scores to evaluate modelsscikit-learn user guideValidation and learning curves, and how to read bias and variance from them.
Classical Machine Learning overviewSpotted a mistake? Tell us.