Explain statistical significance in copy A/B tests and why one day fails.

WHAT THIS TESTS: The interviewer wants to know if you understand that A/B testing is a randomized experiment governed by statistical hypothesis testing, not a simple comparison of raw numbers. They are checking whether you can separate true signal from random noise, and whether you appreciate why sample size and time matter for valid inference.

A GOOD ANSWER COVERS: First, define statistical significance as the probability that an observed difference in conversion rates between variant A and variant B reflects a genuine effect rather than random fluctuation. Second, explain that a one-day sample is usually too small to achieve adequate statistical power, and that daily traffic patterns, day-of-week effects, and external events can temporarily skew results. Third, mention that proper A/B tests require a predetermined sample size or run time, a chosen significance level such as 0.05, and ideally a power analysis to control false negatives. Fourth, note that stopping early and picking the leader inflates the false positive rate because random early leads regress toward the mean as more data arrives.

COMMON WRONG ANSWERS: Claiming that the higher conversion rate is automatically the winner regardless of sample size or variance. Saying that statistical significance is just a fancy way to say the numbers look big. Arguing that copy tests do not need rigor because words are cheap to change. Proposing to run the test until one version wins without accounting for peeking or multiple comparison problems.

LIKELY FOLLOW-UPS: How would you calculate the required sample size before launching the test? What would you do if the test reaches significance but the absolute lift is tiny? How do you handle seasonality or traffic spikes during a test? Can you explain the difference between statistical significance and practical significance?

ONE CONCRETE EXAMPLE: Imagine variant A has a 2.0 percent conversion rate and variant B has a 2.4 percent conversion rate after one day with only 200 visitors per variant. The 0.4 percentage point gap looks promising, but with such a small sample the confidence interval likely spans zero and the result is not statistically significant. If you ship B based on that single day, you might be chasing noise. After running for two weeks to reach ten thousand visitors per variant, the same 0.4 percentage point gap might hold with a p-value below 0.05, giving you confidence the copy change actually moved the needle.

Source: Wikipedia: A/B testing