Histograms versus summaries for latency

WHAT THIS TESTS Whether you understand the fundamental difference: a summary computes quantiles client-side at observation time, while a histogram defers quantile computation to query time using bucket counts.

A GOOD ANSWER COVERS A histogram divides the value range into predefined buckets and counts observations per bucket. Because bucket counts are additive, you can sum the buckets from many instances and compute a fleet-wide approximate quantile at query time, for example with histogram_quantile in PromQL. The accuracy is bounded by bucket width, so quantiles falling in a wide bucket are interpolated and approximate. A summary computes specific quantiles, say p50, p90, p99, locally on each instance over a sliding window. Those quantiles are exact for that instance but cannot be aggregated: there is no valid way to combine per-instance p99 values into a fleet p99. Summaries also fix which quantiles exist at instrumentation time, so you cannot ask for a new one later.

COMMON WRONG ANSWERS Saying you can average or take the max of summary quantiles across servers to get a global quantile; this is statistically invalid. Claiming histograms are always exact; their accuracy depends on bucket boundaries. Forgetting that summaries are more CPU-expensive client-side.

LIKELY FOLLOW-UPS How do exponential or native histograms improve on explicit buckets? How do you choose bucket boundaries? What is the storage cost of many buckets times many label combinations?

ONE CONCRETE EXAMPLE A fleet of 50 API servers each exports p99 latency as a summary. The on-call engineer needs the overall p99 but cannot get it: averaging the 50 p99 values understates the true tail. Switching to histograms lets them sum http_request_duration_seconds_bucket across all instances and apply histogram_quantile(0.99, ...) for a correct, if bucket-bounded, fleet-wide tail latency.