Why use median/p95 for API latency instead of the mean?

WHAT THIS TESTS: This question tests your practical understanding of statistics as applied to system performance. Interviewers want to see that you know latency data is not normally distributed and has a long tail. They are looking for your ability to connect abstract metrics (mean, median, percentile) to the concrete impact on user experience and the reliability of a service. It separates candidates who just know definitions from those who have actually diagnosed and monitored real-world systems.

A GOOD ANSWER COVERS: A strong answer has four parts. First, define the terms: mean is the average, median (p50) is the midpoint of the data, and p95 is the value that 95% of requests are faster than. Second, explain that API latency distributions have a long tail of slow requests caused by garbage collection, network hiccups, or cold starts, which makes the distribution skewed. Third, describe why the mean is misleading: these few extreme outliers drag the mean upwards, giving an unrepresentative view of the typical experience. Fourth, explain why percentiles are better: the median shows the true typical user experience, unaffected by outliers, while p95 quantifies the 'pain point' for your slowest cohort of users, making it a critical metric for setting SLOs.

COMMON WRONG ANSWERS: A major red flag is only defining the statistical terms without connecting them to latency distributions or user experience. Many candidates say 'the mean is sensitive to outliers' but can't explain what causes those outliers in a service (e.g., GC pauses, network packet loss, database contention). Another common mistake is confusing the meaning of p95, incorrectly stating it's the latency for '95% of users' rather than the upper bound for 95% of requests. Finally, a weak answer fails to provide a simple numerical example of a dataset where the mean and median diverge significantly.

LIKELY FOLLOW-UPS: Expect questions like 'When would you care about p99 or p99.9?' (Answer: For highly critical systems like payment processing, where a 1% failure or slow-down rate is unacceptable). Another is 'How would you use p95 to set an SLO?' (Answer: Define a target, like '95% of all search queries must complete in under 300ms over a 28-day window'). You might also be asked about the underlying data structures used to calculate percentiles efficiently, like histograms or t-digest sketches.

ONE CONCRETE EXAMPLE: Consider 10 API requests with latencies in milliseconds: 50, 52, 48, 55, 60, 51, 53, 49, 58, and one outlier at 1500. The mean (average) is (526 + 1500) / 10 = 202.6 ms. This number doesn't represent the experience of any user; it's artificially high. The median, however, is found by sorting the data and finding the middle value. The sorted list is [48, 49, 50, 51, 52, 53, 55, 58, 60, 1500]. The median is the average of the 5th and 6th values, (52 + 53) / 2 = 52.5 ms. This median value accurately reflects the typical experience for 9 out of 10 users, while the p95 (or in this small sample, p90+) highlights the terrible experience of the one unlucky user.