Preparing features

Classical Machine Learning, lesson 10

Scale numbers, encode categories and fill gaps inside one pipeline.

"Regularisation with L1 and L2" (penalties on weights), "Logistic regression", "Decision trees", and "Cross-validation and tuning" (a pipeline refitted in each fold).

Part 1. Units change the answer

Three customers. A pays $60 a month and filed no support tickets in the last 90 days. B pays $64 and filed four. C pays $70 and filed none. Which two are most alike? For churn, B is the odd one out: four tickets in three months is a customer with a problem.

A method that measures likeness by straight-line distance, as k-means does in "Clustering and PCA", takes each column's numbers as they come.

Example 1.1

Distance before and after scaling

A to B√(4² + 4²)=5.7
A to C√(10² + 0²)=10.0
In raw units, B looks closer to A than C does. A $4 difference counts as much as four extra tickets.
billspread $30
ticketsspread 1
Across all customers, bills spread about $30 around their average and tickets about 1. One ticket is a much bigger difference than one dollar.
A to B√((4 ÷ 30)² + (4 ÷ 1)²)=4.0
A to C√((10 ÷ 30)² + 0²)=0.33
Divide each difference by its column's spread first. Now C is the close one and B is far away, which is what you'd expect.

Penalised models have the same problem. "Regularisation with L1 and L2" charges for the size of each weight, and a weight's size depends on units: bill in cents needs a weight a hundred times smaller than bill in dollars, so its L2 penalty is ten thousand times smaller.

Standardisation

Rescaling a column to average 0 and spread 1: subtract the column's mean and divide by its standard deviation, both measured on the training rows. A $95 bill, when the average is $65 and the spread is $30, becomes 1.0. scikit-learn calls it StandardScaler.

Min-max scaling maps the smallest value to 0 and the largest to 1: bills from $10 to $120 put $95 at (95 − 10) ÷ 110 = 0.77. One extreme value squeezes everyone else into a narrow band, so standardisation is the usual default.

Trees are the exception. A split only asks whether a value is above or below a threshold, so any rescaling that keeps the order (dollars to cents, or even a log) gives the same splits and the same predictions.

Your turn

You change the bill column from dollars to cents and refit each model with the same settings. Whose predictions stay exactly the same?

The tree finds the same splits at thresholds 100 times larger, so it sorts every customer exactly as before. k-means measures distance, which bill in cents now dominates. Penalised logistic regression charges the new, tiny bill weight almost nothing, so the penalty stops holding bill back while still holding back every other column.

Show the answer

D. The tree finds the same splits at thresholds 100 times larger, so it sorts every customer exactly as before. k-means measures distance, which bill in cents now dominates. Penalised logistic regression charges the new, tiny bill weight almost nothing, so the penalty stops holding bill back while still holding back every other column.

Part 2. Categories as numbers

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

Categories as numbers
Gaps and awkward shapes
One pipeline for all of it

See Pro plans I have Pro: sign in

Next lessonData leakageFind the columns and steps that let the answer slip into training. Start lesson 11

Classical Machine Learning overviewSpotted a mistake? Tell us.

10 · Preparing features