More in Analytics & Metrics — page 10
Explain time series stationarity and how to achieve it
Tests your grasp of core time series assumptions. Define stationarity (constant mean/variance over time), explain why models need it for stable predictions, and name methods to test and achieve it. A red flag is just saying the data looks 'flat'.
How would you measure a sales forecast model's accuracy?
Tests if you can link statistical metrics to business outcomes. Define MAE (average error) and RMSE (penalizes large errors). Choose RMSE when large misses are costly (e.g., stock-outs), MAE otherwise. A red flag is reciting formulas without business context.
How would you analyze a time series of user sign-ups?
This tests your structured approach to decomposing time series data. A strong answer identifies trend (long-term growth), seasonality (weekly/yearly patterns), and irregular components like spikes or dips.
How CUPED increases statistical power in experiments
Tests your grasp of variance reduction in A/B testing. Explain how CUPED uses correlated pre-experiment data to reduce outcome variance, increasing statistical power. A red flag is confusing it with simpler difference scores, which can actually increase noise.
Why is user-level randomization flawed by spillover effects?
Tests your grasp of SUTVA violations in network experiments. Explain how spillover contaminates the control group, then propose graph cluster randomization—grouping users and assigning entire clusters to A/B variants—to minimize interference.
Primary vs. Guardrail Metrics in Experiments
Tests your grasp of risk management in A/B testing. A great answer defines a primary metric as the goal and a guardrail as a 'do no harm' check. A feature ships only if the primary improves without hurting guardrails.
What is the novelty effect in experimentation?
This tests your grasp of temporary user behavior changes that can invalidate A/B tests. A strong answer defines the effect, explains how it inflates metrics, and suggests running tests longer or segmenting by user tenure. A red flag is ignoring mitigation.
Why is stopping an A/B test early problematic?
Tests understanding of the 'peeking problem' in A/B testing. A good answer defines peeking, explains how it inflates false positive rates, and contrasts it with waiting for a pre-determined sample size. A red flag is not explaining the statistical mechanism.
How do you determine sample size and duration for an A/B test?
This tests your grasp of statistical power and business trade-offs. A good answer defines the four inputs (baseline, MDE, significance, power) to calculate sample size, then uses traffic to find duration.
How would you A/B test a 'Buy Now' button color change?
Tests your grasp of the A/B testing lifecycle. A strong answer defines a clear hypothesis (e.g., 'a green button will increase clicks'), selects a primary metric (CTR), and considers guardrail metrics. A red flag is skipping the hypothesis and metrics.
Why can't we t-test p99 latency in an A/B test?
This tests if you know why t-tests fail for percentiles. A t-test requires a normally distributed statistic (like the mean), but a sample p99's distribution isn't normal.
Explain Simpson's Paradox with a user engagement example
Tests if you see beyond aggregate data. Define the paradox, give a numerical example where a feature fails overall but wins in segments (e.g., new vs. returning users), and name the confounding variable. A vague definition without numbers is a red flag.
DAU dropped 10% overnight. Is this a significant change?
Tests your use of statistical hypothesis testing on business metrics. Outline the process: state a null hypothesis (no change), choose a Z-test, calculate the p-value, and compare to an alpha of 0.05. A red flag is guessing causes before proving significance.
Explain correlation vs. causation with a software example
This tests your critical thinking about data and ability to avoid logical fallacies. A good answer defines both terms, then gives a software example where a third, confounding variable (like traffic) is the true cause of two correlated metrics.
A/B test p-value is 0.08, PM wants to ship. What now?
Tests if you can translate statistical risk into business terms for a PM. A good answer defines the 8% false positive risk, weighs it against the cost of shipping, and suggests next steps like running the test longer instead of just saying no.
What does a p-value of 0.03 mean in an A/B test?
This tests your grasp of statistical significance and ability to make a data-driven decision. A great answer defines p-value, compares it to alpha (0.03 < 0.05), and concludes to reject the null hypothesis. A red flag is misstating the p-value's definition.
Mean vs. Median for API Response Times?
Tests your understanding of non-normal distributions like latency. Choose median as it's robust to outliers that skew the mean. Then, state that even median is insufficient; percentiles (p99, max) are crucial for capturing the full user experience.
Design a Fast, Multi-Dimensional Analytics Dashboard
This tests your grasp of data warehousing trade-offs for analytics. A great answer compares OLAP cubes (fast, pre-computed) vs. columnar DBs (flexible, real-time), then proposes a hybrid model.
How would you validate a feature's success, correcting for self-selection?
This tests your ability to separate correlation from causation. A great answer identifies self-selection bias, proposes a quasi-experimental method like Propensity Score Matching to create a control group, and then compares outcomes.
A/B Test Results with Skewed Traffic: What's Next?
This tests your ability to spot confounding variables. A good answer invalidates the results due to sampling bias, proposes segmenting the data by device to find the true effect, and suggests re-running the test with correct randomization.