Cause and effect

Statistics and Regression, lesson 13

Why a correlation isn't a cause, and how to get closer to one.

A regression shows that two things move together, and whether one moves the other depends on how the data came about. This lesson splits a measured difference into the real effect and the bias, then removes the bias with a coin or with a control. You need regression slopes and standard errors.

By the end, you can:

Split a measured difference into the effect and selection bias.
Explain why random assignment removes selection bias, and what it leaves.
Remove a known confounder with a control, and work out the bias it caused.

Part 1. Two costs, one seen

Split a measured difference into the effect and selection bias.

A desk wants to know if its new execution algorithm trades more cheaply than the old one. A trader chose which orders got it. Every order has two costs, one per algorithm, and you only ever see one.

Example 1.1

A saving of 8 basis points

Orders that got the new algorithm cost 5 basis points (bp, hundredths of a percent), and orders on the old one cost 13 bp. A simulation, where you can see both costs, says the new group's orders would have cost 7 bp on the old algorithm. How much did the algorithm save?

The plain difference13 − 5 = 8 bp
What a report would show.
The new group's two costs7 − 5 = 2 bp
The same orders, with and without the new algorithm. That's the effect.
The rest13 − 7 = 6 bp
On the old algorithm, the old group's orders cost 6 bp more than the new group's would have: they were harder orders.
Put together8 = 2 + 6
Take away the unseen 7 and add it back: (13 − 7) + (7 − 5).

Answer2 bp. The other 6 bp of the measured saving came from which orders the trader picked.

The plain difference compares two different sets of orders. Here the trader gave the new algorithm small orders on calm days, which are cheap with any algorithm. So the gap measured the algorithm plus the difference between the orders, and nothing in the data alone says how to split it.

Figure 1Each order has a cost on each algorithm, but you see only one. The orange cell, the new group's cost on the old algorithm, is never seen. Without it, 8 bp can't be split into 2 of effect and 6 of selection bias.

Selection bias

When the cases that got a treatment differ from the rest in ways that change the outcome, the plain difference mixes the treatment's effect with that difference. The mixed-in part is selection bias. The unseen outcome, what would have happened otherwise, is the counterfactual.

measured = effect + selection bias: 8 = 2 + 6

Selection bias doesn't shrink with more data. A thousand orders picked the same way give the same 8 bp with a smaller standard error, so you become sure of the wrong number. A small p-value says the gap isn't luck. It can't say where the gap came from.

Your turn

Orders on a new algorithm cost 6 bp and orders on the old one 15 bp. A simulation says the new group's orders would have cost 10 bp on the old algorithm. What's the effect, and what's the selection bias?

The effect compares the same orders: 10 − 6 = 4 bp. The bias is how much more the old group's orders cost on the same algorithm: 15 − 10 = 5 bp. Together they make the plain difference, 9 bp.

Show the answer

B. The effect compares the same orders: 10 − 6 = 4 bp. The bias is how much more the old group's orders cost on the same algorithm: 15 − 10 = 5 bp. Together they make the plain difference, 9 bp.

Part 2. Let a coin choose

Explain why random assignment removes selection bias, and what it leaves.

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

Let a coin choose
When you can't flip a coin

See Pro plans I have Pro: sign in

The last lesson in Statistics and RegressionBack to the overviewLearn from data without fooling yourself Back to Statistics and Regression

Statistics and Regression overviewSpotted a mistake? Tell us.

13 · Cause and effect