Fixing watermelon SLOs that hide unhappy users
Whether SLIs truly track user experience.
Green-outside, red-inside SLOs come from server-side measurement, coarse aggregation, wrong events, and lax thresholds; fix by measuring at the client and using meaningful SLIs.
WHAT THIS TESTS This probes whether you can critically evaluate SLI design and recognize when a metric has decoupled from the experience it is supposed to represent.
A GOOD ANSWER COVERS A watermelon SLO is green on the outside and red on the inside: dashboards meet target while users complain. The technical causes are measurement and aggregation flaws. Measuring server-side omits client-perceived latency, network issues, DNS, and edge failures, so the user feels pain the server never records. Over-aggregation hides localized failure: averaging across all endpoints, regions, or tenants lets one broken endpoint or one bad region disappear into a healthy global number. Counting the wrong events, such as treating a fast error page as a success, inflates the SLI. Relying on averages instead of percentiles hides the slow tail. And thresholds set too generously, like a five-second latency target, technically pass while users perceive the service as broken.
THE FIX Move measurement as close to the user as possible, ideally real-user monitoring or client-side telemetry. Segment SLIs by critical user journey, region, and cohort so localized pain surfaces. Use percentiles, not means, for latency. Define success as a genuinely good outcome, excluding fast failures. Tighten thresholds to match real expectations, and validate SLIs against actual support tickets and user feedback.
COMMON WRONG ANSWERS Concluding the users are mistaken or overly sensitive rather than questioning the metric. Adding more dashboards without fixing where or what you measure. Keeping global averages that mathematically guarantee local problems vanish.
LIKELY FOLLOW-UPS How does real-user monitoring differ from synthetic? Why do percentiles beat averages here? How granular should segmentation be before it becomes noise? How do you reconcile SLIs with qualitative feedback?
ONE CONCRETE EXAMPLE A global API reports 99.95 percent availability, yet users in one region are furious. The SLI averaged across all regions, so a sustained outage affecting five percent of users in a single region barely moved the global number. You re-cut the SLO per region and per critical journey, switch latency to p99 measured from the client, and immediately the affected region shows red, matching the complaints. The dashboard now tells the truth, and the team can act.
Read the original → sre.google
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.