Why can't we t-test p99 latency in an A/B test?

WHAT THIS TESTS: This question probes your understanding of the assumptions behind common statistical tests. It specifically tests whether you know that the Central Limit Theorem (CLT) applies to the sample mean, but not necessarily to other statistics like percentiles. Answering well demonstrates statistical maturity beyond simply defaulting to a t-test for all comparisons. It assesses your ability to identify when a standard parametric test is inappropriate and propose a valid non-parametric alternative like bootstrapping.

A GOOD ANSWER COVERS: First, explain why a t-test is invalid. A t-test compares the means of two groups and assumes the sampling distribution of the mean is normal, a guarantee provided by the CLT for large samples. However, you have only one p99 value from each group, and the sampling distribution of a single percentile is not guaranteed to be normal. Second, propose bootstrapping as a non-parametric method that makes no assumptions about the statistic's distribution. Third, describe the process: 1) For B iterations (e.g., 10,000), create a bootstrap sample for both control and treatment by drawing N values with replacement from the original latency data. 2) In each iteration, calculate the p99 of each bootstrap sample and record the difference. 3) After B iterations, you have a distribution of p99 differences. Fourth, explain how to interpret the result: construct a 95% confidence interval by taking the 2.5th and 97.5th percentiles of your distribution of differences. If this interval does not contain zero, the result is statistically significant.

COMMON WRONG ANSWERS: "You can't use a t-test because latency data isn't normal." This is a weak answer. The t-test is quite robust to non-normal data for comparing means due to the CLT, especially with large samples. The critical error is applying it to a percentile. Another red flag is calculating the p99 for many small sub-samples and then running a t-test on those p99 values; this is statistically invalid. Finally, just saying "use bootstrapping" without explaining the mechanism or why it's necessary shows superficial knowledge.

LIKELY FOLLOW-UPS: "How would you choose the number of bootstrap iterations?" (Answer: 1,000 is a minimum, but 10,000 is common for stable confidence intervals. You need enough samples to reliably estimate the tails of the distribution.) "What if bootstrapping is too computationally expensive for our dataset?" (Answer: Acknowledge the trade-off. Mention alternatives like a permutation test or, for very large datasets, analytical approximations based on the order statistics, though this is more advanced.)

ONE CONCRETE EXAMPLE: Imagine our control and treatment groups each have 50,000 latency measurements. The observed p99 is 450ms for control and 420ms for treatment, an observed difference of -30ms. We run 10,000 bootstrap iterations. In each, we resample 50k latencies with replacement from control and 50k from treatment, calculate the p99s, and store the difference. After sorting the 10,000 differences, we find the 2.5th percentile is -38ms and the 97.5th percentile is -22ms. Our 95% confidence interval for the true difference is [-38ms, -22ms]. Since this interval does not contain zero, we can conclude the treatment caused a statistically significant reduction in p99 latency.