Experimentation
189 bites tagged Experimentation — interview questions with model answers, and 60-second explainers.
HEART Framework: Five Metrics for Product Health
HEART is a vital-signs panel for product health, not a single score. Use it to pick metrics for feature launches so you track user value, not vanity numbers. Measuring all five dimensions when only one matters creates dashboard bloat and analysis paralysis.
PPC: Buying Intent, Not Attention
PPC buys intent, not attention: you bid to appear when someone searches for what you sell, paying only if they click. Use it for high-intent offers like B2B software where organic reach is slow.
What is Simpson's Paradox and how can it bias A/B tests?
Tests whether you recognize that aggregate trends can reverse within subgroups. A strong answer defines the paradox, gives an A/B example where treatment wins overall but loses in every segment due to skewed allocation, and prescribes stratified analysis.
How do network effects violate A/B tests and how to mitigate them?
Tests SUTVA violations and network experiment design. Answers note treated users alter control outcomes, then propose social-graph cluster randomization to isolate spillovers. Red flag: ignoring peer-to-peer spillover and using user-level randomization.
A/B test: 0.1% lift. Statistical vs practical significance?
Statistical significance says the 0.1% is real; practical significance asks if revenue exceeds engineering cost. Frame with CIs and ROI. Distinguishing proof from business value. Shipping just because p is low while ignoring cost.
How do you determine required sample size for an A/B test?
Tests statistical power and experimental design. Name four inputs: baseline conversion rate, minimum detectable effect, alpha (5%), and power (80%), then solve for N. Red flag: "test until significant" or fixed guesses like 1000 users without effect size.
How do you determine sample size for a conversion lift experiment?
Tests fluency with statistical experiment design. A strong answer frames N as a function of alpha, power, baseline rate, and MDE, noting that shrinking the MDE or raising power inflates N. Red flag: picking N from traffic instead of risk tolerance.
What are your null and alternative hypotheses for this A/B test?
This tests translating a directional business question into statistical hypotheses. A strong answer states H0 as no difference in registration rate and H1 as green outperforming blue. A red flag is framing H0 as "blue is better" or using a two-tailed test.
How would you recommend launching a checkout flow with mixed A/B metrics?
This tests multi-metric trade-offs. A strong answer tags conversion as success and AOV as a guardrail, estimates net revenue impact, and frames decision as a risk-managed business choice. A red flag is demanding all metrics win or ignoring business context.
Build a reusable A/B framework and its copy API
Propose a declarative hook with experiment key, fallback, and variants; handle bucketing, anti-flicker, and logging. API design and separation of concerns in experimentation.
How does A/B testing native mobile apps differ from web?
Native mobile constraints versus web deployment. A strong answer covers: app store cycles forcing remote flags; device fragmentation skewing segments; and offline usage delaying events. Assuming mobile mirrors web with instant updates.
What are the engineering challenges and biases in overlapping A/B tests?
This tests inference across concurrent experiments. A strong answer names interaction effects and collision bias, proposes mutual exclusion or MVT for coupled UI, and stratified bucketing. Red flag: assuming additive lifts without combination validation.
Design an A/B test system for homepage headlines without deployments
Tests decoupling experiment configuration from deploys via a headless CMS and feature flagging. A strong answer covers variation containers, server-side bucketing, conversion tracking, and marketer-controlled copy swaps.
How would you track which headline wins in an A/B test?
Tests end-to-end experiment instrumentation across the stack. Outline: deterministically bucket users, serve variant A or B, emit click events, and aggregate by variant.
How do you diagnose a confounded A/B test with slower page load?
This tests confounding beyond p-values. A strong answer isolates the bug, checks if slower load hurt or helped conversions, and judges whether copy or latency caused the lift. A red flag is defending a 5% win because p < 0.05 while ignoring randomization.
Compare A/B/n testing with multi-armed bandits for headline optimization
Contrast A/B/n with dynamic MAB allocation; A/B/n optimizes final inference, MABs optimize reward; flag real-time infra. Grasp of explore-exploit and when rigor beats speed. Calling MABs always better ignoring regret.
How does cookie clearing affect A/B results and consistency?
This tests bucketing integrity when state changes re-randomize users. Explain that re-bucketing contaminates the sample and triggers SRM; propose deterministic server-side assignment, fingerprinting, or authenticated IDs.
Explain statistical significance in copy A/B tests and why one day fails.
This checks if you distinguish signal from noise. A strong answer defines statistical significance as confidence a difference is real, warns that one-day samples are small and skewed by variance, and cites false positive risk.
How do you implement a CTA A/B test and attribute conversions?
This tests experiment architecture from bucketing to attribution. A strong answer covers: stable user bucketing, server or client-side rendering, and conversion events tagged with experiment and variant IDs.
Architect email subject line A/B testing for a large user base
Tests distributed systems and statistical rigor. Covers deterministic user bucketing, isolated variant delivery, deduplicated tracking, and pre-powered significance. Red flag: daily re-randomization or using open rate without confidence intervals.
Design a system to A/B test headlines for a single article URL
Hash users for sticky variants; store separately; emit events; compute CTR. controlled experiment design with user bucketing and attribution. client-side randomization without stickiness or event tracking.
Design a centralized Copy Service with versioning, segmentation, and experiments
This tests separation of editorial workflow from runtime delivery. A strong answer covers immutable versioned records, a resolution API evaluating segmentation rules, and delegating experiment bucketing externally.
Architect a real-time multi-armed bandit and compare trade-offs to A/B testing
Sketch a fast arm router, streaming feedback, and model updates; contrast MAB regret minimization with A/B's unbiased estimates. Real-time ML serving and statistical trade-offs.
Describe how you'd implement an A/B test for a landing page headline
Mention deterministic bucketing, anti-flicker rendering, tracking with variant IDs, and sample-size planning. Experiment integrity beyond DOM swaps. Swapping headlines client-side after paint biases data and hurts UX.
Get Experimentation bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.