How to Statistically Test a 10% DAU Drop?

WHAT THIS TESTS: This question assesses your ability to apply a rigorous, quantitative framework to a common business problem. It's not just about knowing the definition of a p-value; it's about demonstrating you can use statistical tools to distinguish a real signal (a significant change) from noise (random fluctuation) to guide business decisions. The interviewer wants to see a structured, methodical approach, not just intuition.

A GOOD ANSWER COVERS: A strong answer walks through the formal steps of hypothesis testing. First, state the null hypothesis (H₀) that the mean DAU has not changed and the alternative hypothesis (H₁) that it has decreased. Second, choose a statistical test. A two-sample t-test or a Z-test is appropriate here. You'd use a Z-test if you have a large sample size (e.g., >30 days of historical data) and can assume you know the population standard deviation. Otherwise, a t-test is more robust. Third, define your significance level (alpha), typically 0.05 or 0.01, which is the probability of rejecting the null hypothesis when it's true. Fourth, calculate the test statistic and the corresponding p-value. The p-value represents the probability of observing a drop this large or larger, assuming the null hypothesis is true. Finally, compare the p-value to alpha. If p < alpha, you reject the null hypothesis and conclude the drop is statistically significant.

COMMON WRONG ANSWERS: A major red flag is answering based on intuition alone, like "A 10% drop is huge, so it's definitely significant." This completely misses the point of the question. Another weak answer is just naming a test ("I'd run a t-test") without explaining the null hypothesis, the significance level, or how to interpret the p-value. Confusing statistical significance with practical significance is also a mistake; a tiny drop could be statistically significant with enough data, but not practically important. Finally, jumping straight to causes ("It was probably the new feature we shipped") before establishing significance is a classic error. First, confirm if there's a fire, then look for the smoke.

LIKELY FOLLOW-UPS: "What if you only have 5 days of historical data? What test would you use then?" (Answer: A t-test is more appropriate due to the small sample size). "What does a p-value of 0.03 actually mean in this context?" (Answer: It means there's a 3% chance of seeing a DAU drop of this magnitude or greater purely by random chance, assuming the true mean DAU hasn't changed). "The drop is statistically significant. What are the first 3 things you would investigate?" (Answer: Check for data pipeline errors, look at segmentation by platform/region/user type, and correlate the drop with any recent code deploys or external events).

ONE CONCRETE EXAMPLE: Let's say our historical daily average DAU is 1,000,000 with a standard deviation of 20,000. Today's DAU is 900,000. Our null hypothesis is that the true mean is still 1,000,000. The observed value is (900,000 - 1,000,000) / 20,000 = -5 standard deviations from the mean. The probability of observing a result 5 standard deviations away from the mean is incredibly small (about 1 in 3.5 million), so the p-value would be <<< 0.05. We would confidently reject the null hypothesis and conclude the drop is statistically significant.