insideout Get Pro

02 · Confidence intervals

Free

Statistics and Regression, lesson 2

Confidence intervals

A range that usually contains the truth, and what usually means.

An estimate on its own hides how far off it might be. A confidence interval shows it: the estimate, give or take about two standard errors. You need the standard error from the last lesson, and the bell curve from Probability.

By the end, you can:

  1. Build a 95% confidence interval for an average.
  2. Say what a 95% confidence level promises, and what it doesn't.
  3. Work out whether an interval rules out zero, and how much data would.

Part 1. The estimate, give or take two standard errors

Build a 95% confidence interval for an average.

The last lesson ended with five days of GULL averaging 0.5%, give or take about 0.9%. "Give or take" can be made exact. Here's the recipe on a bigger sample.

Example 1.1

A 95% interval from 25 days

25 days of GULL average +0.30%, with a sample standard deviation of 2.0%. Give a 95% confidence interval for GULL's true daily average.

  1. Find the standard error2.0% / √25 = 0.40%

    s / √n, from the last lesson.

  2. Find the half-width1.96 × 0.40% = 0.78%

    1.96 standard errors. Where 1.96 comes from is below.

  3. Go that far either way0.30% ± 0.78%

    Read it as "0.30%, give or take 0.78%".

Answer−0.48% to +1.08% a day.

Where does 1.96 come from? An average of many days is close to bell-shaped, by the central limit theorem from Probability. A bell curve holds 95% of its weight within 1.96 standard deviations of its middle. So 95% of the time, a sample average lands within 1.96 standard errors of the truth.

95%2.5%2.5%−1.96 SEtruth+1.96 SEwhere a sample average lands
Figure 1Where a sample average can land. 95% of the time it's within 1.96 standard errors of the truth, and 2.5% of the time it's further out on each side.

Now turn it around. If the average is within 1.96 standard errors of the truth, the truth is within 1.96 standard errors of the average. So the range x̄ ± 1.96 SE catches the truth in 95% of samples.

Confidence interval

A confidence interval is a range built from a sample by a recipe that catches the true value a set share of the time. That share is the confidence level. For an average, at 95%:

x̄ ± 1.96 × s / √n

Other levels use other numbers: 1.645 for 90% and 2.576 for 99%. More confidence costs width. From the same 25 days, the 99% interval is 0.30% ± 1.03%.

With few points, s is itself a rough guess, and 1.96 is a little too small. Careful work uses a number from the t distribution instead: 2.06 for 25 points, 2.26 for 10. At 100 points it's 1.98, close enough to 1.96.

Your turn

100 days of a stock average +0.2%, with a sample standard deviation of 1.5%. What's the 95% confidence interval for its true daily average?

SE = 1.5% / √100 = 0.15%, and 1.96 × 0.15% = 0.29%, so 0.2% ± 0.29%. Dividing by 100 instead of √100 gives the too-narrow +0.17% to +0.23%. Using s in place of the SE gives a range for single days. Leaving out the 1.96 gives +0.05% to +0.35%, only a 68% interval.

Part 2. What the 95% means

Say what a 95% confidence level promises, and what it doesn't.

The 95% is easy to misread. The clearest way to see what it means is to run the recipe many times on a stock whose truth you know.

Example 2.1

A hundred intervals

GULL's true average is 0.1% a day. Draw 100 samples of 25 days, build a 95% interval from each, and count how many contain 0.1%.

  1. Build one interval per samplex̄ ± 2.06 × s / √25

    The same recipe each time, with the t number for 25 points. New days each time.

  2. See where they land100 different intervals

    Each centre moves with its sample's average, and each width moves with its s.

  3. Count the hits95 of 100 contain 0.1%

    The 5 misses came from samples that happened to land far from the truth.

Answer95 of the 100 intervals caught the truth. Another run might catch 93 or 97.

−2%−1%0%+1%+2%truth0.1%95 of 100 catch it.5 miss, in orange.
Figure 2A hundred 95% intervals, each from 25 days of GULL. 95 cross the true average. The 5 in orange miss it, and nothing about those samples looked wrong.

The truth is fixed. Once you've built your interval, it either contains the truth or it doesn't, and there's no chance left about it. The 95% belongs to the recipe: run it on many samples, and about 95 in 100 of its ranges catch the truth.

Confidence level

The share of samples for which the interval recipe catches the truth. It describes the recipe, not any one interval. "This recipe works 95% of the time" is right. "There's a 95% chance the truth is in this range" is the common misreading.

Two more misreadings. A 95% interval for the average says nothing about single days: GULL's days spread over several percent, far wider than any interval for its average. And a new sample's average lands in your interval only about 83% of the time, not 95%, because both samples wobble.

Your turn

Your 95% interval for GULL's average is −0.2% to +0.6% a day. Which statement is right?

The true average is fixed. It's either in this range or it isn't, and the 95% belongs to the recipe. B mixes up the spread of days with the spread of averages. D is off too: a new average lands in this range only about 83% of the time, because both samples wobble.

Part 3. Narrower intervals cost data

Work out whether an interval rules out zero, and how much data would.

The half-width of an interval is 1.96 × σ / √n, so intervals shrink the way standard errors do: slowly. For a strategy, the question that matters is usually whether its interval is clear of zero.

Example 3.1

Can you rule out zero?

A strategy averaged +0.08% a day over 400 days, with a daily standard deviation of 1.2%. Give a 95% interval for its true daily average. Does it rule out zero? If not, how many days would, if the average held at 0.08%?

  1. Find the standard error1.2% / √400 = 0.06%

    √400 is 20.

  2. Find the half-width1.96 × 0.06% = 0.118%

    1.96 standard errors either way.

  3. Build the interval0.08% ± 0.118% = −0.038% to +0.198%

    It includes zero.

  4. Solve for the days1.96 × 1.2% / √n < 0.08% → √n > 29.4 → n > 864

    The half-width has to shrink below the average itself.

Answer−0.038% to +0.198%, so zero is still possible. About 865 days, three and a half years, would clear it.

zero: no edge−0.1%0+0.1%+0.2%+0.3%100 days400 days865 days1,600 daysaverage 0.08%
Figure 395% intervals around a 0.08% average with a 1.2% daily spread. At 400 days the interval still crosses zero. At 865 it just reaches it, and at 1,600 it sits clear.

Margin of error

Half the width of a confidence interval is its margin of error. At 95% it's 1.96 standard errors, so it shrinks with √n: a third of the width takes nine times the data.

1.96 × 1.2% / √400 = 0.118%

An interval that contains zero doesn't show the edge is zero. This one also contains +0.19%, which would be a large edge. It says only that 400 days can't tell those apart. "No evidence of an edge" and "evidence of no edge" are different claims.

Your turn

A strategy averages 0.1% a day with a daily standard deviation of 2%. If the average holds, about how many days before its 95% interval sits clear of zero?

You need 1.96 × 2% / √n < 0.1%, so √n > 39.2 and n > 1,537, about six years. 39 forgets to square. 400 leaves out the 1.96. 784 squares the 20 but multiplies by 1.96 only once.

Where you'll meet it

In interviews

"What does a 95% confidence interval mean?" is a staple of quant and data science interviews, and the trap is saying there's a 95% chance the truth is inside this particular interval. Say that the 95% belongs to the recipe. Follow-ups ask how the width scales with n, and when to use the t distribution instead of 1.96.

On the job

Opinion polls report a margin of error, half the width of a 95% interval. For a poll of 1,000 people split near 50/50, the standard error of the share is √(0.5 × 0.5 / 1,000) = 1.6%, and 1.96 × 1.6% gives the familiar "plus or minus 3 points". It covers the luck of who was asked, not people who refuse to answer.

Checking the 95%, in Python
import random
import statistics

random.seed(2)
mu, sigma, n = 0.10, 2.0, 25
t = 2.064  # the t value for 95% with 25 points (1.96 would be a little narrow)

hits = 0
for _ in range(10_000):
    xs = [random.gauss(mu, sigma) for _ in range(n)]
    m, s = statistics.fmean(xs), statistics.stdev(xs)
    half = t * s / n ** 0.5
    hits += m - half <= mu <= m + half
print(f"{hits / 10_000:.1%} of the intervals contain the true mean")

# Part 3: 400 days, average 0.08%, standard deviation 1.2%
se = 1.2 / 400 ** 0.5
print(f"95% interval: {0.08 - 1.96 * se:+.3f}% to {0.08 + 1.96 * se:+.3f}%")

# Output:
# 94.7% of the intervals contain the true mean
# 95% interval: -0.038% to +0.198%

Check yourself

Four questions from across the lesson. Try each one before you look back.

1

Your interval for a mean runs from 0.2 to 0.8. You want one a third as wide, at the same confidence. How much more data do you need?

Width is proportional to 1 / √n. A third of the width needs √n three times bigger, so n nine times bigger.

2

A desk builds a 90% interval for the average return of each of 50 strategies. About how many of the intervals miss their strategy's true average?

A 90% recipe misses about 10% of the time: 50 × 0.10 = 5. More data makes each interval narrower, but the share that misses stays the same.

3

From the same data you build a 90% interval and a 99% interval. How do they compare?

The widths follow the multipliers: 2.576 / 1.645 = 1.57. More confidence costs width. Nine times is the data a third of the width takes, a different question.

4

You have 10 data points and want a 95% interval for their average. Which multiplier should go in front of s / √n?

With 10 points, s is a rough guess, so the t distribution's number for 10 points, 2.26, replaces 1.96. 1.645 gives a 90% interval and 2.576 a 99% one.

What to remember

  1. 1The estimate, give or take two standard errors

    A 95% interval for an average is x̄ ± 1.96 × s / √n: 25 days averaging 0.30% with s = 2.0% give −0.48% to +1.08%.

  2. 1The estimate, give or take two standard errors

    With few points, swap 1.96 for the t number: 2.06 for 25 points, 2.26 for 10.

  3. 2What the 95% means

    The 95% belongs to the recipe: about 95 intervals in 100 catch the truth, and any one interval either does or doesn't.

  4. 3Narrower intervals cost data

    The margin of error shrinks with √n: 0.08% a day on a 1.2% spread needs about 865 days before its interval clears zero.

Next lessonTests and p-valuesNext, the same question turned around: if the truth were zero, how surprising would your data be? That number is a p-value. Start lesson 3

Statistics and Regression overviewSpotted a mistake? Tell us.