Design an experiment and logging to link API latency to engagement

WHAT THIS TESTS: The interviewer wants to see if you can move beyond dashboard correlation and build a causal argument. They are looking for three specific competencies: designing a valid randomized controlled experiment, instrumenting a distributed system so server-side performance data can be reliably joined with client-side behavioral data, and reasoning about statistical power and confounders at scale.

A GOOD ANSWER COVERS: First, define the hypothesis and primary metric. State that the hypothesis is that added API latency reduces items viewed per session, and pick a guardrail metric like error rate so you do not harm reliability. Second, describe the randomization mechanism. Explain that you would bucket users or sessions randomly at the edge or API gateway into control and treatment groups, ensuring that the treatment group receives artificially injected latency while the control group receives baseline performance. Third, detail the logging contract. Propose that every API response includes a unique trace ID or request ID in a response header, and that the client analytics payload echoes that same ID when it fires engagement events like item viewed or session heartbeat. Fourth, explain the join strategy. Describe landing both streams, server performance logs and client analytics events, into a common data warehouse, then joining on the shared trace ID with a time bounded window to handle clock skew. Fifth, mention analysis. Note that you would compute the difference in means for items viewed per session between the two groups and use a statistical test to check significance, rather than just comparing averages.

COMMON WRONG ANSWERS: A major red flag is proposing to analyze historical observational data by regressing engagement on response time. This fails because user segments, device types, and network conditions confound the relationship; slow responses may correlate with heavy users on poor networks rather than causing disengagement. Another red flag is suggesting a join on user ID plus timestamp without a shared request identifier, which produces a Cartesian product and inflates row counts. A third red flag is ignoring sample size and power, leading to an experiment that runs for weeks without a detectable effect.

LIKELY FOLLOW-UPS: The interviewer may ask how you would handle dropped client events when the network is slow. A strong answer discusses using sessionization and probabilistic attribution or comparing per session metrics only for sessions with at least one ping. They may also ask how you would roll this out safely. Mention canary deployments, automatic shutoffs if error rates spike, and stratified randomization to preserve cohort balance. They might also probe how you would detect novelty effects versus true latency sensitivity by running the experiment for multiple business cycles.

ONE CONCRETE EXAMPLE: Imagine an e-commerce search API. You add a sleep middleware that injects 200 milliseconds of latency for the treatment group. The API response header includes X-Request-ID. The mobile client reads this header and attaches it to every analytics event fired during that session. In your warehouse, you join the server log row, which has request ID and latency, to the client event rows sharing that request ID. You aggregate to the session level, summing items viewed, then compare the treatment distribution to the control using a t-test. If the treatment mean is 4.2 items and the control mean is 5.1 items with a p-value below 0.05, you have evidence that latency causes the drop.

Source: optimizely.com