How do you diagnose a confounded A/B test with slower page load?

WHAT THIS TESTS: This tests your ability to separate statistical significance from causal validity in online experiments. The interviewer wants to see if you treat A/B tests as systems that can break, not just as formulas that output p-values. You need to demonstrate skepticism toward the randomization mechanism, understand that implementation bugs create confounders, and know how to reason about composite treatments.

A GOOD ANSWER COVERS: A good answer hits four things in order. First, stop treating the p-value as the final word; a significant result in a broken test is meaningless. Second, diagnose the confound by verifying that the 20% slower page load was truly an implementation artifact and not a deliberate part of the new copy experience. Third, reason about the direction of bias: slower load times usually depress conversions, so if the treatment still won despite worse performance, the true copy effect is likely larger than 5%; conversely, if the latency somehow helped, the copy effect could be zero or negative. Fourth, propose a remediation path such as re-running the test with fixed loading parity or using a stratified analysis if you can segment users by experienced load time.

COMMON WRONG ANSWERS: The most common wrong answer is declaring the test valid because p is less than 0.05 while dismissing the latency gap as a minor detail. Another red flag is assuming the 20% slowdown automatically invalidates the test without reasoning through the causal direction. A third wrong pattern is suggesting you simply subtract the 5% lift from some imagined latency penalty; conversion impacts do not combine linearly like that.

LIKELY FOLLOW-UPS: Interviewers often push deeper by asking how you would salvage the data without re-running the test, whether you would use a linear model or CUPED to control for load time, or how you would design the next experiment to prevent this. They may also ask what you would do if the slower load was caused by the new copy itself, turning latency into a mediating variable rather than a confounder.

ONE CONCRETE EXAMPLE: Imagine the control loads in 1.0 second and the treatment loads in 1.2 seconds. Past data shows every 100ms increase drops conversion by roughly 1%. If the treatment still shows a 5% raw lift, the copy itself might actually deliver a 7% lift after accounting for the roughly 2% drag from latency. You would present this as a directional estimate, insist on a clean rerun for the final decision, and add a guardrail metric for page load time in the test configuration.

Source: Statsig Blog