Confidence Interval: Quantifying Uncertainty in Your Estimates

The mental model

A confidence interval (CI) quantifies the uncertainty around an estimate. Instead of giving a single number (a point estimate) like "average user session is 10 minutes," a CI provides a range, such as "we are 95% confident the true average user session is between 8 and 12 minutes." It acknowledges that your measurement, based on a sample, is probably not the exact true value for the entire population. The CI gives you a plausible range for that true value.

How it works

You start with a sample from a larger population, like 1,000 user visits out of millions. You calculate a statistic from that sample, like the average time on page. Then, using properties of the sample (like its size and standard deviation), you construct a range around that statistic. The "confidence level," typically 95%, is a property of the procedure. It means that if you were to repeat the sampling process many times, 95% of the confidence intervals you calculate would contain the true population parameter.

When to use it

Confidence intervals are essential whenever you report a metric calculated from a sample, not the entire population. This is common in three areas: first, in A/B testing to determine if a change had a statistically significant effect and to understand the magnitude of that effect; second, in political polling or market research surveys to report results with a margin of error; third, in performance monitoring when you sample system metrics like latency instead of recording every single request.

The footgun

The most common misinterpretation is thinking a 95% CI means there is a 95% probability that the true population parameter lies within this specific interval. This is incorrect from a frequentist perspective. The 95% refers to the long-run success rate of the method used to create the intervals. Imagine 100 different teams running the same experiment; the method ensures that about 95 of their calculated CIs will capture the true value. Your specific interval either contains the true value or it doesn't.

One canonical example

An A/B test shows a new checkout button (Variant B) has a 5.5% conversion rate, while the old button (Variant A) has 5.0%. The point estimate is a +0.5% lift. A confidence interval might state: "We are 95% confident that the true lift for Variant B is between +0.1% and +0.9%." This tells you not only that the effect is positive (the interval is entirely above zero) but also gives a plausible range for how big the improvement really is.