How does CUPED increase the statistical power of an experiment?

What's really being asked

This question probes your practical knowledge of running sensitive experiments. The interviewer is testing if you understand the mechanisms of variance reduction beyond just a buzzword. They want to see if you can articulate how to increase an experiment's power (its ability to detect a true effect) without simply increasing sample size or duration, which have direct business costs. It separates candidates who have only read about A/B testing from those who have dealt with its real-world limitations, like detecting small but meaningful effects.

The full answer

First, define CUPED as a statistical technique that increases power by reducing the variance of an outcome metric. It's like noise-canceling headphones for your data. Second, explain the mechanism: it is a large-scale application of ANCOVA (Analysis of Covariance). It uses a pre-experiment covariate (X), which is highly correlated with the in-experiment outcome metric (Y), to predict and subtract the 'expected' or 'baseline' portion of the outcome. Third, state the data requirement: you need the same metric for each user from both a pre-experiment period and the experiment period itself. Fourth, quantify the impact: the variance of the adjusted metric is reduced by a factor of (1 - rho^2), where rho is the correlation between the pre-experiment and in-experiment metric. A higher correlation means a greater reduction in variance.

The mistakes people make

A major red flag is confusing CUPED with a simple difference-in-differences or 'change score' (Y - X) approach. This simpler method is only effective if the correlation is > 0.5 and can actually increase variance and reduce power if correlation is low. A senior candidate must know that CUPED's regression-based adjustment is superior because it is never worse than the unadjusted metric. Another weak answer is vaguely saying 'it reduces noise' without explaining the role of correlation, the ANCOVA model, or the (1 - rho^2) formula.

What usually comes next

Be ready for 'When would you NOT use CUPED?' (Answer: When you have no pre-experiment data, or when the correlation is near zero, making the gain negligible for the effort). Also, 'How does this reduction in variance translate to business impact?' (Answer: Since required sample size is proportional to variance, a 64% variance reduction means you only need 36% of the original sample size, drastically cutting experiment duration and cost).

A concrete example

To test a feature for increasing daily user engagement, you expect a small 1% lift. A standard test might need 1,000,000 users over 4 weeks. With CUPED, you'd use engagement data from the week before the experiment (X) as a covariate. If the correlation (rho) between pre-experiment and in-experiment engagement is 0.8, you reduce variance by (1 - 0.8^2) = 64%. This means you now only need about 360,000 users to achieve the same statistical power, potentially getting a result in 10 days instead of 28.