How CUPED increases statistical power in experiments
Tests your grasp of variance reduction in A/B testing. Explain how CUPED uses correlated pre-experiment data to reduce outcome variance, increasing statistical power. A red flag is confusing it with simpler difference scores, which can actually increase noise.
What's really being asked
This question tests your understanding of statistical power and its relationship to variance. It assesses your knowledge of practical, advanced statistical methods used in large-scale experimentation platforms. The interviewer wants to see if you can articulate the 'why' behind the technique—its business implications like faster experiments or detecting smaller effects—not just the 'what'. It also probes your awareness of statistical nuances, like why CUPED is superior to simpler methods like difference scores.
The full answer
First, the core intuition: CUPED is like noise-canceling for your experiment. It uses pre-experiment data to subtract out a user's predictable 'background noise,' making the 'signal' from the treatment easier to detect. Second, the mechanism: It requires a covariate (X), typically the same metric measured in a pre-experiment period, that is correlated with the in-experiment outcome (Y). Third, the math: CUPED creates an adjusted metric whose variance is reduced by a factor of approximately (1 - ρ²), where ρ is the correlation between X and Y. This directly increases statistical power. Fourth, the comparison: Mention that it's a form of ANCOVA and is superior to a simple difference score (Y-X), which can actually increase variance if the correlation is less than 0.5.
The mistakes people make
A major red flag is confusing CUPED with difference-in-differences. A candidate might say 'you just subtract the pre-period value from the post-period value.' This is incorrect because this simpler method is suboptimal and can harm power. The key insight of CUPED is its regression-based adjustment, not simple subtraction. Another common mistake is a vague explanation like 'it uses old data to make it better' without explaining the mechanism of variance reduction via correlation. Finally, not knowing what data is required—the technique is useless without a strongly correlated, pre-randomization covariate.
What usually comes next
When would CUPED not be effective? (When the correlation between the pre-experiment covariate and the outcome is near zero). What if you have multiple potential covariates? (You could use a model to select the most predictive ones, but must be careful about overfitting). Can you implement this in SQL? (This tests practical application; the answer involves joining experiment data with pre-experiment data). How much variance reduction is typical in practice? (A 50% reduction is not uncommon for stable metrics).
A concrete example
Imagine an e-commerce site testing a new checkout button to increase average revenue per user (ARPU). To use CUPED, we would first calculate each user's ARPU for the 28 days before the experiment starts (the covariate, X). During the experiment, we measure each user's ARPU (the outcome, Y). Let's say we find the correlation (ρ) between pre-experiment ARPU and in-experiment ARPU is 0.8. CUPED creates an adjusted ARPU metric. The variance of this new metric will be reduced by a factor of (1 - 0.8²) = 1 - 0.64, which is a 64% variance reduction. This massive reduction means we can detect a much smaller lift in ARPU with the same number of users or reach significance much faster.
Interview question
Why is the CUPED adjustment generally superior to using a simple difference score (post-period value minus pre-period value) for variance reduction in an A/B test?
- a.The simple difference score can increase variance if correlation is low, while CUPED's regression-based adjustment will not.Correct
- b.The simple difference score is a biased estimator of the treatment effect, while CUPED provides an unbiased estimate.
- c.CUPED requires a shorter pre-experiment data period to be effective compared to a simple difference score.
- d.CUPED corrects for systematic imbalances between the treatment and control groups created during randomization.
Why? this is the answer
CUPED's regression-based adjustment is guaranteed to reduce variance. A simple difference score, however, can harm statistical power by increasing variance if the correlation between the pre- and post-period metric is less than 0.5.
Just read this? Test yourself on what you have been reading.
Read the original → zetyra.com
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on a/b testing — each one lists the topics its interview covers.
See open roles