Advanced interview questions in Analytics & Metrics, page 3
How would you validate a feature's success, correcting for self-selection?
This tests your ability to separate correlation from causation. A great answer identifies self-selection bias, proposes a quasi-experimental method like Propensity Score Matching to create a control group, and then compares outcomes.
How would you validate a PM's claim about feature success?
This tests your grasp of causal inference vs. correlation. A great answer acknowledges self-selection bias, proposes a quasi-experimental method like Propensity Score Matching to create a synthetic control group, and then compares outcomes for the matched…
Describe data architecture for fast multi-dimensional slicing and pre-computation trade-offs
Tests analytical architecture trade-offs between latency and cost. Strong answers propose a hybrid pipeline: columnar warehouse for flexible exploration plus pre-aggregated OLAP cubes for sub-second dashboards.
Design a Fast, Multi-Dimensional Analytics Dashboard
This tests your grasp of data warehousing trade-offs for analytics. A great answer compares OLAP cubes (fast, pre-computed) vs. columnar DBs (flexible, real-time), then proposes a hybrid model.
How would you architect a fast, multi-dimensional analytics dashboard?
This tests your grasp of analytics architectures and trade-offs between pre-computation and real-time querying. A great answer clarifies needs, then proposes a hybrid model using an OLAP cube for core metrics and a columnar store for ad-hoc queries.
Why can't you t-test p99 latency, and what's a valid alternative?
Explain that t-tests target means while p99 variance depends on tail density; propose bootstrap CIs or permutation tests.
Why can't we t-test p99 latency in an A/B test?
This tests if you know why t-tests fail for percentiles. A t-test requires a normally distributed statistic (like the mean), but a sample p99's distribution isn't normal.
Why not t-test p99 latency? Describe a valid alternative.
This tests your grasp of statistical test assumptions. A good answer explains why p99 violates t-test normality, then outlines a resampling method like bootstrapping to build a confidence interval on the *difference* of p99s.
Why is user-level randomization flawed by spillover and what is an alternative?
Network interference. A/B tests violate SUTVA when treatment spills across edges, biasing effects. Cluster randomization uses network clusters and Horvitz-Thompson weighting for unbiased estimates.
Why is user-level randomization flawed by spillover effects?
Tests your grasp of SUTVA violations in network experiments. Explain how spillover contaminates the control group, then propose graph cluster randomization—grouping users and assigning entire clusters to A/B variants—to minimize interference.
Handling spillover effects in social network A/B tests
This tests your grasp of SUTVA violations in networked experiments. A great answer explains how user-level randomization causes spillover, then proposes graph cluster randomization to assign entire communities to treatment or control, minimizing…
Explain how CUPED increases statistical power and required data
Tests ANCOVA variance reduction. Answer: CUPED regresses pre-experiment X on Y, shrinking variance by (1-ρ²); needs pre-randomization prognostic baseline; beats difference scores. Red flag: calling it Y-X subtraction or saying it changes the effect.
How CUPED increases statistical power in experiments
Tests your grasp of variance reduction in A/B testing. Explain how CUPED uses correlated pre-experiment data to reduce outcome variance, increasing statistical power. A red flag is confusing it with simpler difference scores, which can actually increase noise.
How does CUPED increase the statistical power of an experiment?
Tests your grasp of variance reduction. Explain CUPED as ANCOVA, using pre-experiment data (X) to remove predictable noise from the outcome (Y). Effectiveness depends on correlation (rho), reducing variance by (1-rho^2).
Diagnosing model degradation over time
Name it model drift, split data vs concept drift; diagnose by comparing distributions and ruling out pipeline bugs; fix via monitoring and retraining.
How do you diagnose and fix a model's degrading performance?
Tests your MLOps process for handling model decay. Name it "concept drift," then outline a plan: diagnose by comparing data distributions, solve with a targeted retraining strategy, and implement proactive monitoring. A red flag is just saying "retrain it."
How do you handle model performance degradation over time?
This tests MLOps lifecycle awareness. Name concept drift, outline a systematic diagnosis of data and error patterns, discuss retraining strategies, and propose a monitoring plan. A red flag is just saying 'retrain the model' without any diagnosis.
Describe two methods for generating prediction intervals or probabilistic forecasts
Tests uncertainty quantification for risk-adjusted decisions. Strong answers: (1) parametric intervals via forecast error variance and normal multipliers, (2) bootstrap residual resampling for empirical percentiles.
Describe two methods for generating prediction intervals
This tests your grasp of uncertainty quantification. A great answer contrasts an analytical method (assuming normal errors, using multipliers like 1.96 for 95%) with a simulation method (bootstrapping residuals).
Describe two methods for generating prediction intervals
This tests your understanding of forecast uncertainty. Describe two methods: 1) assuming normally distributed errors and using a standard deviation multiplier, and 2) bootstrapping residuals to simulate future paths.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles