Everything in Product Management, page 19
How do you optimize a data warehouse for billions of rows?
Tests physical design in columnar warehouses at scale. Strong answers cover partition and cluster pruning, materialized views or rollups to reduce joins, caching and search indexes for hot paths, and reserved slots or autoscaling.
Describe dbt's role and how it differs from traditional ETL
Position dbt as ELT's T with lineage, tests, docs; contrast with Python ETL using external compute and Airflow only scheduling tasks.
Describe tracking a user event end-to-end from frontend to BI tool
Mention frontend instrumentation, routing, ingestion, warehouse transforms, and BI querying.
Design an experiment and logging to link API latency to engagement
Causal inference and data integration. Randomly inject latency for a treatment group with a control at baseline, then join server trace IDs to client events via a shared request ID. Never confuse correlation with causation or miss join issues.
Design a system to detect sudden add-to-cart drops in real time
This tests streaming pipeline design and seasonality-aware anomaly detection. Outline Kafka or Kinesis ingestion, windowed aggregations, and ML baselines tuned to hourly and weekly trends. Red flag: static thresholds that ignore daily patterns.
How would you design UTM capture and attribution persistence?
Capture UTMs on landing, store in a first-party cookie with TTL, attach to events, and persist on conversion.
What event and data payload track Add to Cart actions?
This tests GA4 ecommerce schema design. Fire add_to_cart with items array containing item_id, price, currency, quantity; include user_id, user_segment, session_id, and timestamp. Red flag: generic button_click with DOM selectors instead of semantic data.
Describe two methods for generating prediction intervals or probabilistic forecasts
Tests uncertainty quantification for risk-adjusted decisions. Strong answers: (1) parametric intervals via forecast error variance and normal multipliers, (2) bootstrap residual resampling for empirical percentiles.
Random split vs walk-forward validation in forecasting
Random splits leak future data into training; walk-forward validation rolls the origin ahead, testing only on later observations.
Describe key components for EDA on three years of daily user sign-ups
This tests time-series decomposition intuition. A strong answer covers trend, seasonality, and noise via plots, autocorrelation, and calendar effects, plus checks for missing days and outliers. Red flag: jumping to forecast models before validating structure.
Explain how CUPED increases statistical power and required data
Tests ANCOVA variance reduction. Answer: CUPED regresses pre-experiment X on Y, shrinking variance by (1-ρ²); needs pre-randomization prognostic baseline; beats difference scores. Red flag: calling it Y-X subtraction or saying it changes the effect.
Why is user-level randomization flawed by spillover and what is an alternative?
Network interference. A/B tests violate SUTVA when treatment spills across edges, biasing effects. Cluster randomization uses network clusters and Horvitz-Thompson weighting for unbiased estimates.
What is the difference between a primary metric and a guardrail metric?
Tests whether you distinguish success criteria from safety checks in experiments. A strong answer defines primary metrics as the target outcome, guardrails as protective thresholds, and gives a concrete scenario where a primary lift does not justify shipping…
What is the 'novelty effect' in experimentation?
Tests whether you separate temporary curiosity from durable value. A strong answer defines novelty effect as short-term behavior change triggered by new elements, notes it inflates early experiment lift, and proposes longer runtimes or lagged cohort analysis.
Why is stopping an A/B test at first significance problematic?
Tests peeking and Type I error inflation. Name peeking; explain daily looks inflate false positive rates above nominal alpha; note p-values assume one look at fixed sample size; recommend pre-committed runtimes or sequential testing.
How do you determine sample size and duration for an A/B test?
This tests statistical power literacy. A strong answer names baseline rate, MDE, alpha, and beta; explains the duration versus sensitivity trade-off; and notes traffic allocation. A red flag is ignoring power or stopping early when results look significant.
Design an A/B test for a 'Buy Now' button color change
Tests structured experiment design from hypothesis to metric. Strong answers: define a falsifiable hypothesis; pick purchase conversion as primary; size the sample and duration; randomize by user; pre-commit to stopping rules.
Why can't you t-test p99 latency, and what's a valid alternative?
Explain that t-tests target means while p99 variance depends on tail density; propose bootstrap CIs or permutation tests.
Explain Simpson's Paradox and construct a user engagement scenario
Tests whether you spot trends reversing when population mixes differ. Good answers define the paradox, give a numerical example with per-segment wins but aggregate loss, and warn against segment-only decisions.
Determine if a 10% DAU drop is statistically significant
Tests signal vs noise in stable metrics. Good answers define a null hypothesis, compute a test statistic from historical variance, compare to a critical value at set alpha, and check seasonality. Red flag: calling a large drop real without baseline variance.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles