insideout Get Pro

04 · Regularisation with L1 and L2

Pro

Classical Machine Learning, lesson 4

Regularisation with L1 and L2

Charge the model for big weights, and see which ones shrink or vanish.

"Linear regression" (coefficients and mean squared error) and "Training, validation and test sets" (overfitting, and choosing on validation rows).

Part 1. Wild weights

Go back to the degree-11 curve from "Training, validation and test sets", the one that passed through all 12 points. Solve for its weights exactly, with x running from 0 to 1, and you find +24 million, −29 million and +22 million side by side. The degree-3 curve's largest weight is about 27.

Huge weights of opposite sign cancel almost perfectly at the 12 training points. Between them, a small change in x upsets the balance and the curve swings. That's what overfitting looks like from the inside.

Near-copy columns do the same thing in an ordinary table. Say two listing sites were merged: one gives each flat's area in m², the other in square feet, rounded to the nearest 10. The two columns carry the same information, apart from the rounding.

Your turn

A plain least-squares fit on 20 such flats gives −212 dollars per m² and +21.6 dollars per square foot. What does it predict for a flat 1 m² (10.76 sq ft) bigger than another?

Add the two effects: −212 + 21.6 × 10.76 ≈ +20.5. The weights are huge and opposed, and together they give back a sensible $20 per m². The fit used the rounding between the two columns to shave a little off its training error. On a new flat whose square feet were rounded the other way, those weights move the prediction by about $216 for every 10 sq ft of rounding.

Both problems have the same cure: make big weights cost something. Instead of fitting to make the error as small as possible, fit to make the error plus a charge on the size of the weights as small as possible.

Example 1.1

Two scores added together

  1. training error+α × size of the weights

    The fit now balances two things: matching the training rows, and keeping the weights small. α (alpha) sets how much the second one counts.

  2. α = 0plain least squares

    With α = 0 the charge vanishes, and you get the plain fit, wild weights and all.

  3. α hugeevery weight near 0→predict the average

    With a huge α, any weight costs more than it saves, so the model flattens out and predicts the average every time: underfitting.

  4. α in betweenweights only as big as the data justifies

    In between, a weight has to earn its size by cutting a lot of error. You find that α by trying several and scoring each on validation rows.

Regularisation

Adding a charge for large weights to the score a model is fitted on, so it prefers simpler answers. Its strength, α, is a setting you choose, not a number the fit learns.

Part 2. L2: ridge

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. L2: ridge
  2. L1: lasso
  3. Using a penalty well
See Pro plansI have Pro: sign in
Next lessonLogistic regressionTurn a weighted sum into a probability, then choose where to say yes. Start lesson 5

Classical Machine Learning overviewSpotted a mistake? Tell us.