Mean or median for API response times?

WHAT THIS TESTS: Whether you understand that latency distributions are not normal or Gaussian. Interviewers want to see that you recognize API response times are right-skewed, multi-modal, and dominated by hiccups like GC pauses, hypervisor stalls, database reindexing, or cache flushes. They are checking if you know why central tendency alone fails for operational metrics and why tail latency determines real user experience more than typical behavior does.

A GOOD ANSWER COVERS: First, state that median is the better central tendency metric because the mean is easily skewed by outliers. A single ten-second GC pause among a thousand fifty-millisecond requests can push the mean from fifty milliseconds to several hundred milliseconds, making it lie about typical performance. Second, immediately qualify that median is still insufficient because it discards the upper half of the distribution entirely and tells you nothing about worst-case behavior. Third, explain that latency is heavily multi-modal due to hiccups, so the only honest way to report it is through percentiles. Fourth, advocate for tracking p95, p99, and especially the maximum, noting that with request fanout users frequently hit tail latencies on every page load. Fifth, mention that averaging percentiles across dimensions or time windows is mathematically invalid and that many dashboards do exactly this.

COMMON WRONG ANSWERS: Saying mean is acceptable because it is easy to compute or because your data looks normal. Claiming median alone is good enough for SLAs and user-facing dashboards. Dismissing the maximum as noise rather than signal. Suggesting standard deviation is meaningful for latency. Proposing to average percentiles across hosts or intervals. Saying you only need the ninety-fifth percentile because it captures the common case.

LIKELY FOLLOW-UPS: How many nines matter for your service and why. How request fanout increases the probability of experiencing tail latency. Why averaging percentiles is mathematically absurd. What tooling mistakes you see in dashboards like Grafana. How you would set an SLO based on latency. Whether you would ever use a histogram or heatmap instead of a line graph.

ONE CONCRETE EXAMPLE: Consider a web page that triggers one hundred API requests. If the median latency is fifty milliseconds but the ninety-ninth percentile is two seconds, the probability that a user experiences at least one two-second request is roughly one minus zero point nine nine to the hundredth power, or about sixty-three percent. Reporting only the median fifty milliseconds would hide the fact that most users actually suffer a visible tail latency. This is why Gil Tene argues that looking only at the common case is a marketing system that dupes engineers and customers into thinking the system is healthy when it is not.

Source: bravenewgeek.com