Bootstrapping: Quantifying Uncertainty with Resampling

THE MENTAL MODEL: Bootstrapping is like pulling yourself up by your own bootstraps. With only one data sample, you can estimate the uncertainty of a statistic by treating that sample as the entire universe. You repeatedly draw new samples from it to simulate what might happen if you could run your experiment many times over, all without collecting new real-world data.

HOW IT WORKS: The process uses resampling with replacement. Imagine you have a sample of 100 user session durations. To create one bootstrap sample, you randomly draw 100 times from your original sample, putting each selected value back before the next draw. Some original values will be chosen multiple times, others not at all. You then calculate your statistic (e.g., the median) on this new bootstrap sample. By repeating this process thousands of times, you generate a distribution of your statistic, from which you can derive a confidence interval or standard error.

WHEN TO USE IT: Use bootstrapping when you need to understand the uncertainty of an estimator, but the underlying theoretical distribution is unknown or complex. It is the standard method for calculating confidence intervals for statistics like the median, percentiles, or the ratio of two variables. It's also useful with smaller sample sizes where assumptions for traditional parametric tests might not hold.

WHEN NOT TO USE IT: Do not use bootstrapping if your original sample is not representative of the population. Bootstrapping cannot fix or detect sample bias; it will only reflect the characteristics of the data it's given. It also performs poorly for statistics that are determined by a small number of extreme values, such as the sample maximum or minimum, as these may not appear in many resamples.

ONE CANONICAL EXAMPLE: Suppose you have five server latency measurements: [12ms, 15ms, 18ms, 25ms, 100ms]. You want a 95% confidence interval for the median latency. The median is 18ms, but how certain is this estimate? You create thousands of bootstrap samples by drawing five values with replacement. For instance, one sample might be [12, 12, 18, 25, 25], with a median of 18ms. Another might be [15, 18, 100, 100, 100], with a median of 100ms. After generating 10,000 such medians, you sort them. The values at the 2.5th percentile (250th value) and 97.5th percentile (9750th value) form your 95% confidence interval.