Why not t-test p99 latency? Describe a valid alternative.

WHAT THIS TESTS: This question tests your ability to choose the right tool for a high-stakes engineering problem. Interviewers want to see if you understand the assumptions behind common statistical tools (like the t-test) and know what to do when those assumptions are violated. It specifically probes your knowledge of non-parametric methods and why they're essential for metrics with non-normal distributions, like latency percentiles.

A GOOD ANSWER COVERS: A strong answer has three parts. First, explain why a t-test is invalid: the Central Limit Theorem applies to the mean, not to percentiles. The sampling distribution of a percentile like p99 is not Gaussian, which is a core assumption for the t-test. Second, describe a valid resampling method. For bootstrapping, you would repeatedly sample with replacement from your control and treatment latency data, calculate the p99 of each resample, find the difference, and repeat this thousands of times (e.g., 10,000). Third, explain how to interpret the result. This process gives you a distribution of p99 differences. You can then calculate a 95% confidence interval from this distribution. If this interval does not contain zero, you can declare a statistically significant difference.

COMMON WRONG ANSWERS: The most common red flag is suggesting a fix that's also wrong, like log-transforming the p99 values themselves. The p99 is a single number; you transform the raw data, not the summary statistic. Another mistake is just saying "the data isn't normal" without explaining why that matters for the t-test (which relies on the sampling distribution of the statistic being normal). A weak answer might mention bootstrapping but be unable to explain the "sample with replacement" mechanic or how to use the resulting distribution to make a decision.

LIKELY FOLLOW-UPS: "How would you choose the number of bootstrap iterations? What's the trade-off?" (Answer: More is better for stability, but computationally expensive. 10,000 is a common starting point. You're trading CPU time for a more stable confidence interval). "What if bootstrapping is too slow? Are there alternatives?" (Answer: Mention quantile regression or methods based on order statistics, but acknowledge they are more complex). "The reference mentions a permutation test. How is that different?" (Answer: Permutation tests simulate the null hypothesis to get a p-value. Bootstrapping estimates the confidence interval of the effect size. For A/B testing, their conclusions are often identical.)

ONE CONCRETE EXAMPLE: Let's say control p99 is 500ms and treatment is 480ms. We run 10,000 bootstrap iterations. In each, we resample with replacement from the original control and treatment latencies, calculate each group's p99, and record the difference. We sort these 10,000 differences. To get a 95% confidence interval, we look at the 250th value (2.5 percentile) and the 9,750th value (97.5 percentile). If our interval is [-25ms, -5ms], it doesn't contain zero, so we're 95% confident the treatment is a real improvement. If the interval is [-25ms, +5ms], it contains zero, and we can't conclude there's a difference.