Determine if a 10% DAU drop is statistically significant

What's really being asked

This question tests whether you treat metrics as random variables rather than deterministic constants. Interviewers want to see that you understand stable metrics still have variance, and that a percentage drop is meaningless without knowing the baseline distribution. The core skill is applying statistical inference to business data and resisting panic-driven incident response.

The full answer

A good answer hits four things in order. First, define the null hypothesis that the observed DAU is drawn from the same distribution as historical stable days. Second, quantify the baseline by estimating the mean and standard deviation from a representative window of prior days, ideally thirty to ninety days after removing known anomalies. Third, choose a test statistic such as a z-score or t-score and compute it from the observed value, then either compare that statistic to a critical value or evaluate the resulting p-value against a predetermined alpha such as zero point zero five. Fourth, control for confounders before testing, specifically checking for day-of-week effects, holidays, or data pipeline delays that could explain the drop without an underlying user behavior change. Only after these steps would you reject the null hypothesis and declare a significant change.

The mistakes people make

Common wrong answers include saying ten percent is obviously significant because the number is big, or comparing to an arbitrary threshold like five percent without data. Another red flag is ignoring the time-series nature of DAU and assuming each day is an independent draw. Some candidates propose complex models like ARIMA without first doing a simple variance check, or they jump straight to segmentation before establishing whether the top-line movement is even real.

What usually comes next

Interviewers often push deeper by asking how you would handle multiple comparisons if you are monitoring hundreds of metrics simultaneously. They may ask what you would do if the variance itself is non-stationary, or how you would balance false positives against the cost of a slow incident response. Another common follow-up is to ask how you would design an automated alerting system that encodes this statistical rigor.

A concrete example

Suppose your stable DAU averages one million with a standard deviation of twenty thousand. A ten percent overnight drop means you observed nine hundred thousand. The z-score is negative five, which far exceeds the critical value of roughly plus or minus one point nine six for a two-tailed test at alpha equals zero point zero five. The p-value would be effectively zero. However, if the drop happened on the Sunday after Thanksgiving, you would first compare against the prior four Sundays rather than the full ninety-day average, because the null hypothesis must reflect the correct baseline.