How would you design an experiment measuring API latency impact on retention?

WHAT THIS TESTS: This question evaluates whether you can construct a causal experiment rather than rely on correlation. The interviewer wants to see if you understand user-level randomization, know how to inject latency technically without breaking functionality, and can pick metrics that reveal tail effects rather than hiding them in averages. It also checks your ability to connect a systems-level change to a business outcome like retention.

A GOOD ANSWER COVERS: First, randomization at the user ID level so that every request from a given user lands in the same bucket, preventing mixed experiences that dilute the treatment effect. Second, a clear technical implementation such as adding deterministic sleep or throttling at the API gateway or middleware based on the user's experiment bucket; the delay should be consistent per user and applied server-side to avoid client variability. Third, metric selection that pairs a primary business metric like D7 or D30 retention with secondary latency metrics including mean, median, and upper quantiles such as P95 and P99. Fourth, analysis using quantile testing to compare the full distribution of latencies across groups, because a treatment might rescue tail experiences without moving the mean. Fifth, guardrail metrics like error rate and session length to ensure the latency injection does not degrade stability or trigger timeouts.

COMMON WRONG ANSWERS: A major red flag is proposing to analyze historical slow requests versus fast requests without randomization, which invites confounding variables like device type or network quality. Another is injecting latency client-side or only on certain endpoints, creating an inconsistent user experience that makes retention impossible to attribute. Relying solely on mean latency is also weak because it masks outliers; a P99 of two seconds can coexist with a mean of two hundred milliseconds, and those tail users are often the ones who churn.

LIKELY FOLLOW-UPS: The interviewer may ask how you would handle network effects if the API is backed by a shared cache or database, since synthetic slowdowns in treatment could spill over and hurt control users. They might also ask how long you would run the experiment to detect a retention signal, which is typically weeks rather than days because retention is a lagging indicator. Another follow-up is how you would segment results by user cohort or device tier to see if latency sensitivity differs across populations.

ONE CONCRETE EXAMPLE: Suppose your control group hits the origin server directly with a P99 API latency of one thousand four hundred sixty milliseconds and a D7 retention rate of sixty percent. Your treatment group is routed through a new edge cache implementation that drops P99 to four hundred sixty four milliseconds, while the mean only falls from two hundred thirty nine milliseconds to one hundred ninety nine milliseconds. Quantile metrics reveal the tail improved dramatically, and D7 retention rises to sixty five percent. Without quantile testing you might have only seen the forty millisecond mean improvement and missed the causal link between tail latency and retention.

Source: GrowthBook Blog, "Measuring A/B test impacts on website latency: using quantile metrics in GrowthBook" (May 21, 2024)