DAU dropped 10% overnight. Is this a significant change?

WHAT THIS TESTS: This question tests your ability to apply a rigorous, scientific framework to a common business problem. Interviewers want to see if you can separate signal from noise using statistical hypothesis testing before jumping into a root-cause analysis. It shows you're data-driven, not just reactive. They are evaluating your understanding of concepts like null hypothesis, p-values, and statistical significance (alpha).

A GOOD ANSWER COVERS: A strong answer walks through the four steps of hypothesis testing. First, formulate the hypotheses: the null hypothesis (H₀) is that the mean DAU has not changed and the observed drop is due to random variance. The alternative hypothesis (H₁) is that the mean DAU has decreased. Second, choose a statistical test. For a large sample size like DAU (typically >30), a two-sample Z-test is appropriate to compare the means of two independent populations (yesterday's users vs. the preceding period's users). Third, calculate the test statistic (Z-score) and the corresponding p-value. The p-value represents the probability of observing a drop this large or larger if the null hypothesis were true. Fourth, make a decision. Compare the p-value to a predetermined significance level (alpha), usually 0.05. If p < 0.05, you reject the null hypothesis and conclude the drop is statistically significant. If p >= 0.05, you fail to reject the null hypothesis, meaning you don't have enough evidence to say the drop isn't just random noise.

COMMON WRONG ANSWERS: The biggest red flag is immediately brainstorming causes for the drop ("Was there a bad code push? Is it a holiday? Did a competitor launch something?"). This skips the most critical first step: verifying that a real, statistically significant event has even occurred. Another weak answer is being vague about the statistical method, saying "I'd see if it's a statistical anomaly" without naming a specific test (like a Z-test or t-test) or mentioning concepts like p-values or a significance level (alpha). Simply saying "a 10% drop is big, so it's significant" is a major red flag, as it ignores the metric's historical variance. A 10% drop for a metric with 1% daily volatility is very different from a 10% drop for a metric with 15% daily volatility.

LIKELY FOLLOW-UPS: "What if you don't have enough historical data to calculate a stable mean and standard deviation?" (Answer: You might need to use a non-parametric test, a t-test for smaller samples, or simply wait for more data). "Let's say it is significant. What are the first three dashboards you would check to find the root cause?" (Answer: Segmentation dashboards: by geography, platform [iOS/Android/Web], and user acquisition channel). "What would you choose for your alpha value, and why?" (Answer: Typically 0.05, which balances the risk of false positives and false negatives. A lower alpha like 0.01 could be used for critical systems to be more certain, but it increases the chance of missing a real issue).

ONE CONCRETE EXAMPLE: Let's say our mean DAU for the last 30 days was 1,000,000 with a standard deviation of 20,000. Yesterday's DAU was 900,000. The null hypothesis is that the true mean is still 1,000,000. The 100,000 user drop is a 5-sigma event (100,000 / 20,000 = 5). The probability of a 5-sigma event occurring by random chance is about 1 in 3.5 million. This would result in an extremely small p-value (much less than 0.05). Therefore, you would confidently reject the null hypothesis and declare the drop statistically significant, triggering an immediate investigation.