Statistics and Regression, lesson 9
Regularisation
Ridge and lasso: shrinking a model so it generalises.
Many signals and little data let least squares fit noise, as the last lesson showed. Regularisation charges a price for large coefficients, so a model fits its own data a little worse and new data better. You need least squares and the bias-variance trade-off.
By the end, you can:
Part 1. Ridge shrinks every coefficient
Compute a ridge coefficient and say what λ does to it.
Ridge regression minimises the squared misses plus λ times the squared coefficients, where λ, read "lambda", sets the price. With one standardised signal, and y measured from its average, there's no intercept to fit, and ridge has a formula you can read.
Example 1.1
One signal, three prices
A standardised signal over 100 rows has Σx² = 100 and Σxy = 40. Find its coefficient by least squares, then by ridge with λ = 25 and λ = 100.
- Least squares40 / 100 = 0.40
Σxy / Σx², the slope from Linear regression with both averages at zero.
- Ridge, λ = 2540 / (100 + 25) = 0.32
λ joins the bottom of the fraction, and nothing joins the top.
- Ridge, λ = 10040 / (100 + 100) = 0.20
When λ equals Σx², the coefficient halves.
Answer0.40, 0.32 and 0.20: the coefficient shrinks towards zero as λ grows, and never quite reaches it.
The penalty acts like one extra row with x = √λ and y = 0. At λ = 25 that's a row at x = 5 insisting y is 0, which pulls the line towards flat. It adds 25 to Σx² and nothing to Σxy, so the bigger λ, the smaller the coefficient.
Regularisation and ridge
Regularisation adds a penalty for complexity to the error a model minimises. Ridge regression penalises the sum of squared coefficients. At λ = 0 it's least squares, and as λ grows every coefficient shrinks towards zero. ŷ, read "y hat", is the model's prediction.
minimise Σ(y − ŷ)² + λΣb²; one signal: b = Σxy / (Σx² + λ)
Ridge earns its keep when signals move together. Two near copies of one signal, correlated 0.997, fitted to 30 rows by least squares get +2.79 and −1.83: one big and positive, one big and negative, nearly cancelling. Ridge with λ = 1 gives +0.69 and +0.24, and λ = 10 gives +0.44 and +0.39. They share.
Give every signal the same spread first. The penalty treats all coefficients alike, and a coefficient's size depends on its signal's units.
Your turn
A standardised signal has Σx² = 200 and Σxy = 30. What's its ridge coefficient at λ = 100?
Part 2. Why a biased answer can win
Show when shrinking an estimate lowers its expected squared error.
Keep going with Pro
You've done the first part. Pro opens the rest of this lesson, and every other one.
- Why a biased answer can win
- Lasso sets some to zero
Statistics and Regression overviewSpotted a mistake? Tell us.