Diagnosing a healthy p50 but breaching p99

WHAT THIS TESTS This evaluates whether you understand tail latency, why averages conceal it, and whether you can run a structured investigation rather than guessing.

A GOOD ANSWER COVERS A stable p50 with a breaching p99 means the typical request is fast but the slowest one percent are not. The user impact is real and often underestimated. Individual users hitting the slow tail experience sluggishness, and in a microservice architecture with fan-out, a single user action may issue dozens of backend calls, so the probability that at least one hits the slow tail rises sharply, dragging up the overall response time most users perceive. To diagnose, first segment: break the p99 down by endpoint, host or pod, tenant, request size, and time of day to see whether the slowness is concentrated or uniform. Then correlate with traces for the slow requests specifically, not the average ones, and look for common tail causes such as garbage collection pauses, lock or mutex contention, thread pool or connection pool exhaustion, cold cache misses, queueing under load, noisy neighbors on shared hardware, retry storms, or a single slow downstream dependency.

COMMON WRONG ANSWERS Dismissing the issue because the average and median are healthy, which ignores the multiplicative effect of fan-out and the disproportionate harm to the heaviest, often most valuable, users. Another error is optimizing the median path, which does nothing for the tail.

LIKELY FOLLOW-UPS How does fan-out mathematically amplify tail latency? How would tail-based trace sampling help here? What is a hedged request? How do you tell a code problem from an infrastructure one?

ONE CONCRETE EXAMPLE A dashboard endpoint shows p50 at forty milliseconds but p99 at two seconds. Segmenting reveals the slow requests cluster on two specific pods. Pulling traces for those slow requests shows long pauses with no downstream activity, pointing at stop-the-world garbage collection on under-provisioned heaps. Right-sizing the heap and tuning the collector collapses the p99 back under target while the median, already fine, is unchanged, confirming the tail, not the typical path, was the problem.