More in Product Management — page 19
Describe tracking a user event end-to-end from frontend to BI tool
WHAT IT TESTS: Grasp of data pipeline layers from product to warehouse. ANSWER OUTLINE: Mention frontend instrumentation, routing, ingestion, warehouse transforms, and BI querying. RED FLAG: Stopping at the backend or confusing analytics with application logs.
Design an experiment and logging to link API latency to engagement
Causal inference and data integration. Randomly inject latency for a treatment group with a control at baseline, then join server trace IDs to client events via a shared request ID. Never confuse correlation with causation or miss join issues.
Design a system to detect sudden add-to-cart drops in real time
This tests streaming pipeline design and seasonality-aware anomaly detection. Outline Kafka or Kinesis ingestion, windowed aggregations, and ML baselines tuned to hourly and weekly trends. Red flag: static thresholds that ignore daily patterns.
How would you design UTM capture and attribution persistence?
WHAT IT TESTS: First-touch attribution from URL to conversion. ANSWER OUTLINE: Capture UTMs on landing, store in a first-party cookie with TTL, attach to events, and persist on conversion. RED FLAG: Last-click without session stitching or ignoring ITP limits.
What event and data payload track Add to Cart actions?
This tests GA4 ecommerce schema design. Fire add_to_cart with items array containing item_id, price, currency, quantity; include user_id, user_segment, session_id, and timestamp. Red flag: generic button_click with DOM selectors instead of semantic data.
Describe two methods for generating prediction intervals or probabilistic forecasts
Tests uncertainty quantification for risk-adjusted decisions. Strong answers: (1) parametric intervals via forecast error variance and normal multipliers, (2) bootstrap residual resampling for empirical percentiles.
Random split vs walk-forward validation in forecasting
WHAT IT TESTS: Awareness of temporal leakage. ANSWER OUTLINE: Random splits leak future data into training; walk-forward validation rolls the origin ahead, testing only on later observations. RED FLAG: Claiming random splits work for time-series.
Describe key components for EDA on three years of daily user sign-ups
This tests time-series decomposition intuition. A strong answer covers trend, seasonality, and noise via plots, autocorrelation, and calendar effects, plus checks for missing days and outliers. Red flag: jumping to forecast models before validating structure.
Explain how CUPED increases statistical power and required data
Tests ANCOVA variance reduction. Answer: CUPED regresses pre-experiment X on Y, shrinking variance by (1-ρ²); needs pre-randomization prognostic baseline; beats difference scores. Red flag: calling it Y-X subtraction or saying it changes the effect.
Why is user-level randomization flawed by spillover and what is an alternative?
TESTS: Network interference. A/B tests violate SUTVA when treatment spills across edges, biasing effects. Cluster randomization uses network clusters and Horvitz-Thompson weighting for unbiased estimates. RED FLAG: Dropping friends or ignoring graph structure.
What is the difference between a primary metric and a guardrail metric?
Tests whether you distinguish success criteria from safety checks in experiments. A strong answer defines primary metrics as the target outcome, guardrails as protective thresholds, and gives a concrete scenario where a primary lift does not justify shipping…
What is the 'novelty effect' in experimentation?
Tests whether you separate temporary curiosity from durable value. A strong answer defines novelty effect as short-term behavior change triggered by new elements, notes it inflates early experiment lift, and proposes longer runtimes or lagged cohort analysis.
Why is stopping an A/B test at first significance problematic?
Tests peeking and Type I error inflation. Name peeking; explain daily looks inflate false positive rates above nominal alpha; note p-values assume one look at fixed sample size; recommend pre-committed runtimes or sequential testing.
How do you determine sample size and duration for an A/B test?
This tests statistical power literacy. A strong answer names baseline rate, MDE, alpha, and beta; explains the duration versus sensitivity trade-off; and notes traffic allocation. A red flag is ignoring power or stopping early when results look significant.
Design an A/B test for a 'Buy Now' button color change
Tests structured experiment design from hypothesis to metric. Strong answers: define a falsifiable hypothesis; pick purchase conversion as primary; size the sample and duration; randomize by user; pre-commit to stopping rules.
Why can't you t-test p99 latency, and what's a valid alternative?
WHAT IT TESTS: Knowing percentiles are order statistics, not means. ANSWER OUTLINE: Explain that t-tests target means while p99 variance depends on tail density; propose bootstrap CIs or permutation tests. RED FLAG: Invoking CLT to justify a t-test on p99.
Explain Simpson's Paradox and construct a user engagement scenario
Tests whether you spot trends reversing when population mixes differ. Good answers define the paradox, give a numerical example with per-segment wins but aggregate loss, and warn against segment-only decisions.
Determine if a 10% DAU drop is statistically significant
Tests signal vs noise in stable metrics. Good answers define a null hypothesis, compute a test statistic from historical variance, compare to a critical value at set alpha, and check seasonality. Red flag: calling a large drop real without baseline variance.
Explain the difference between correlation and causation with a software example.
Tests whether you distinguish association from causation to avoid blaming production issues. A strong answer defines both concepts, names a confounding variable, and gives a software example with a common cause. Red flag: claiming correlation is causation.
A/B test p-value 0.08, PM wants to ship. How do you advise?
Tests statistical rigor versus business pragmatism. A strong answer covers pre-registered thresholds, false positive risk, statistical power, confidence intervals, and the business cost of being wrong. Red flag: shipping without quantifying downside risk.