Why can't you t-test p99 latency, and what's a valid alternative?

WHAT THIS TESTS: This question tests whether you understand the difference between a mean and an order statistic. A standard t-test is derived for the sample mean under assumptions of normality or large-sample behavior via the Central Limit Theorem. The p99 latency is a sample quantile, not an average. Its sampling distribution depends on the probability density near the 99th percentile cutoff, and with only one percent of observations defining the tail, the estimate is noisy and non-normal. Interviewers want to see that you know parametric formulas for standard errors break down here and that you reach for resampling.

A GOOD ANSWER COVERS: Four things in order. First, state explicitly that a t-test compares means and assumes the sampling distribution of the mean is normal or approximately normal. Second, explain that a percentile is an order statistic whose variance is inversely related to the density at that quantile, so in the extreme tail the standard error is large and the distribution can be skewed or discrete. Third, propose a bootstrap method where you draw ten thousand or more resamples with replacement from the control group and the treatment group independently, compute the p99 for each resample, and then construct an empirical confidence interval for the difference in p99 values; if the interval excludes zero, the change is significant. Fourth, mention the permutation test alternative where you pool control and treatment data, randomly reassign labels, and compute the p99 difference under the null hypothesis that both groups share the same distribution.

COMMON WRONG ANSWERS: Claiming that the Central Limit Theorem guarantees the t-test is valid for any statistic given enough data. Applying the t-test directly to the p99 values as if they were a sample mean. Proposing to compare only the tail observations in isolation without preserving the overall sample structure. Suggesting a z-test based on a normal approximation of the quantile without estimating the local density, which is fragile for p99.

LIKELY FOLLOW-UPS: How does required sample size scale when you move from p95 to p99 to p99.9? What would you do if the bootstrap distribution looks bimodal because of a caching layer or timeout cliff? How would you handle dependent observations if the same user generates multiple requests? Can you combine quantile regression with bootstrapping?

ONE CONCRETE EXAMPLE: Suppose control latency has ten thousand observations and treatment has ten thousand. You draw bootstrap resamples of size ten thousand with replacement from each group. For each pair of resamples you calculate the difference in their p99 latencies. After five thousand iterations you find that ninety-five percent of the differences lie between minus two milliseconds and plus fifteen milliseconds. Because the interval does not include zero, you conclude the treatment significantly increased p99 latency.