Analytics &amp; Metrics

Describe key components for EDA on three years of daily user sign-ups

This tests time-series decomposition intuition. A strong answer covers trend, seasonality, and noise via plots, autocorrelation, and calendar effects, plus checks for missing days and outliers. Red flag: jumping to forecast models before validating structure.

Explain how CUPED increases statistical power and required data

Tests ANCOVA variance reduction. Answer: CUPED regresses pre-experiment X on Y, shrinking variance by (1-ρ²); needs pre-randomization prognostic baseline; beats difference scores. Red flag: calling it Y-X subtraction or saying it changes the effect.

Why is user-level randomization flawed by spillover and what is an alternative?

Network interference. A/B tests violate SUTVA when treatment spills across edges, biasing effects. Cluster randomization uses network clusters and Horvitz-Thompson weighting for unbiased estimates.

What is the difference between a primary metric and a guardrail metric?

Tests whether you distinguish success criteria from safety checks in experiments. A strong answer defines primary metrics as the target outcome, guardrails as protective thresholds, and gives a concrete scenario where a primary lift does not justify shipping…

What is the 'novelty effect' in experimentation?

Tests whether you separate temporary curiosity from durable value. A strong answer defines novelty effect as short-term behavior change triggered by new elements, notes it inflates early experiment lift, and proposes longer runtimes or lagged cohort analysis.

Why is stopping an A/B test at first significance problematic?

Tests peeking and Type I error inflation. Name peeking; explain daily looks inflate false positive rates above nominal alpha; note p-values assume one look at fixed sample size; recommend pre-committed runtimes or sequential testing.

How do you determine sample size and duration for an A/B test?

This tests statistical power literacy. A strong answer names baseline rate, MDE, alpha, and beta; explains the duration versus sensitivity trade-off; and notes traffic allocation. A red flag is ignoring power or stopping early when results look significant.

Design an A/B test for a 'Buy Now' button color change

Tests structured experiment design from hypothesis to metric. Strong answers: define a falsifiable hypothesis; pick purchase conversion as primary; size the sample and duration; randomize by user; pre-commit to stopping rules.

Why can't you t-test p99 latency, and what's a valid alternative?

Explain that t-tests target means while p99 variance depends on tail density; propose bootstrap CIs or permutation tests.

Explain Simpson's Paradox and construct a user engagement scenario

Tests whether you spot trends reversing when population mixes differ. Good answers define the paradox, give a numerical example with per-segment wins but aggregate loss, and warn against segment-only decisions.

Determine if a 10% DAU drop is statistically significant

Tests signal vs noise in stable metrics. Good answers define a null hypothesis, compute a test statistic from historical variance, compare to a critical value at set alpha, and check seasonality. Red flag: calling a large drop real without baseline variance.

Explain the difference between correlation and causation with a software example.

Tests whether you distinguish association from causation to avoid blaming production issues. A strong answer defines both concepts, names a confounding variable, and gives a software example with a common cause. Red flag: claiming correlation is causation.

A/B test p-value 0.08, PM wants to ship. How do you advise?

Tests statistical rigor versus business pragmatism. A strong answer covers pre-registered thresholds, false positive risk, statistical power, confidence intervals, and the business cost of being wrong. Red flag: shipping without quantifying downside risk.

What does a p-value of 0.03 mean at alpha 0.05?

0.03 means 3% chance of data this extreme if the null (no effect) holds; since 0.03 < 0.05, reject the null.

Mean or median for API response times?

Tests if you know latency is skewed and outlier-sensitive. Pick median over mean because hiccups distort the mean, but note median hides tail latency. Advocate for p95, p99, and max. Red flag: defending mean as representative or ignoring tail behavior.

Describe data architecture for fast multi-dimensional slicing and pre-computation trade-offs

Tests analytical architecture trade-offs between latency and cost. Strong answers propose a hybrid pipeline: columnar warehouse for flexible exploration plus pre-aggregated OLAP cubes for sub-second dashboards.

How would you validate a feature's conversion impact given self-selection bias?

Tests causal inference for opt-in features. Strong answers use quasi-experiments like propensity matching or diff-in-diff to compare similar users and verify pre-trends. Red flag: a raw t-test between adopters and non-adopters ignoring selection bias.

A/B test shows 5% lift with imbalanced mobile traffic. What next?

This tests sample ratio mismatch and Simpson's paradox. A strong answer says the lift is confounded by platform, demands stratified analysis to isolate the true effect, and checks randomization. A red flag is accepting the 5% lift without questioning skew.

Write SQL to generate a monthly cohort retention table from raw events

Tests window functions and date truncation for cohort analysis. A strong answer finds each user's first month, counts returning users per period, and divides by cohort size. Aggregating all users without isolating acquisition month hides new-user churn.