Statistics and Regression, lesson 8
Overfitting
Train on one part, test on another, and watch complexity fail.
A model with enough knobs can match any data it's given, noise and all, and then fail on the next data. This lesson measures that gap, splits test error into bias and variance, and picks a model by testing it on data it never saw. You need least squares from the last two lessons.
By the end, you can:
Part 1. Training error flatters
Work out how much a model's error on its own data understates its error on new data.
Fit a model, then measure its misses on the same rows. That number is too kind, because the coefficients were chosen to make exactly those misses small. The clearest case is a model fitted to pure noise, with nothing real to find.
Example 1.1
Ten coefficients fitted to noise
You regress 50 months of pure noise, with variance 1, on 9 useless signals plus an intercept: 10 coefficients. What mean squared error do you expect on those 50 months, and on fresh noise at the same signal values?
- Write down the sizesp = 10, n = 50, σ² = 1
No model can beat the noise, so the honest error is 1.
- Error on its own rows1 × (50 − 10) / 50 = 0.80
Each coefficient cancels a little of the noise. Together they absorb about p × σ² of the n × σ².
- Error on fresh noise1 × (1 + 10 / 50) = 1.20
The absorbed noise isn't in the new rows, and the coefficients now add errors of their own.
Answer0.80 on its own rows and 1.20 on new ones, against a true 1: a gap of 0.40.
Think of each coefficient as a knob turned to cancel part of the noise: ten knobs cancel about ten rows' worth. On new rows that noise is gone, and the knobs, still set for it, add error instead. The two errors sit about p / n either side of the truth.
Training error, test error and overfitting
Training error is measured on the rows a model was fitted to, and test error on rows held back. A model much better on its training rows than on test rows is overfitting: it has learned the noise. A model too simple to follow the real shape is underfitting.
training ≈ σ² × (1 − p / n); new rows ≈ σ² × (1 + p / n)
The gap depends on p / n, so data shrinks it: the same 10 coefficients on 500 months flatter by only 2%. The danger is many coefficients on few rows. A hundred fitted to 120 months of noise leave a training error of 1 − 100 / 120 = 0.17, and predict nothing.
Your turn
You regress 100 months of pure noise, variance 1, on 24 useless signals plus an intercept. What error do you expect on the 100 months, and on fresh noise at the same signal values?
Part 2. Too simple, too flexible
Split test error into noise, bias and variance, and find the flexibility that minimises it.
Keep going with Pro
You've done the first part. Pro opens the rest of this lesson, and every other one.
- Too simple, too flexible
- Hold data back
Statistics and Regression overviewSpotted a mistake? Tell us.