Correlate API Slowness with User Engagement

What's really being asked

This question assesses your ability to move beyond simple correlation and design an experiment to prove causation. It tests end-to-end systems thinking: can you design a controlled experiment for a backend metric, architect the data logging and pipeline required to join disparate data sources (performance logs and user analytics), and reason about the statistical validity of the results? Interviewers are looking for a practical, data-driven approach, not just theoretical knowledge of A/B testing.

The full answer

An excellent answer addresses four key areas in order. First, Experiment Design: Propose a controlled A/B test. The control group (A) receives normal API responses. The treatment group (B) receives an artificial, fixed delay, for example, sleep(200ms) in the API middleware. Users should be randomly bucketed into A or B. Second, Data Logging: Explain the critical need for a shared identifier. A unique request_id or session_id must be generated on the client, passed in API headers, logged by the server in its performance logs, and also included in the client-side analytics events. Third, Data Joining: Describe how you would ingest both performance logs (e.g., from Datadog) and user analytics events (e.g., from Segment) into a data warehouse like BigQuery. The join key is the shared ID. Fourth, Analysis: With the unified dataset, you can now group by experiment bucket (A vs. B) and calculate the primary metric (items viewed per session). Mention the need to check for statistical significance (e.g., p-value < 0.05) and calculate the effect size.

The mistakes people make

Proposing a purely observational study is a major red flag. Answering with "let's look at historical data for slow requests and see if those users had lower engagement" only shows correlation and is plagued by confounding variables. For example, users on slow networks might behave differently for reasons other than API latency. Another common mistake is hand-waving the data joining process by saying "we'll just join the logs" without specifying the need for a shared identifier, which is the key technical prerequisite. Finally, failing to mention statistical significance or power analysis suggests a lack of rigor.

What usually comes next

Expect questions about implementation details and trade-offs. For example: "How would you implement the artificial delay without impacting other systems?" or "What are the ethical implications of deliberately slowing the experience for some users?" You might also be asked how long to run the experiment, which opens a discussion on statistical power, weekly business cycles, and collecting a sufficient sample size, typically 2-4 weeks.

A concrete example

Let's say our p95 API response time is 400ms and we hypothesize that responses over 600ms hurt engagement. We can design an A/B test where the 'B' group gets an artificial 250ms delay added to a specific API call. A client app generates a session_id. When it calls the API, it includes this ID in a header. The API server logs the request, including the session_id, the latency, and which bucket ('A' or 'B') it was in. The client app also logs user actions, like 'itemViewed', with the same session_id to an analytics service. In our data warehouse, we join the server logs and analytics events on session_id. We then calculate the average number of items viewed per session for users in bucket A versus bucket B to determine if the added latency had a statistically significant negative impact.