AB Testing
85 bites tagged AB Testing — interview questions with model answers, and 60-second explainers.
Explain statistical power, MDE, and sample size impact
Power is the chance of detecting a true effect; MDE is the smallest lift worth measuring; raising power or shrinking MDE inflates sample size. This checks whether you grasp the power-MDE-sample-size trade-off.
What is a p-value in A/B tests and what does threshold mean?
P-value is the chance of this or more extreme result if the null is true; the 5% threshold is the accepted false-positive rate. your grasp of frequentist testing mechanics. saying it is the probability the variant wins.
How do you set up a button color A/B test?
Tests whether you can define a valid experiment, not just a color change. A strong answer covers hypothesis, randomization, primary metric, sample size, and significance threshold.
When increasing CTA clicks, what side-effects and guardrails should you consider?
This tests balancing growth with business safety. Name guardrails like retention, revenue, fraud, load time; set NI thresholds pre-launch; and include SRM checks. A red flag is treating guardrails as optional success metrics instead of hard stop limits.
Design an A/B test for loss aversion versus gain framing at checkout
Tests whether you can isolate framing effects from checkout confounders. Strong answers detail user-level randomization, event logging, and guardrail metrics like revenue per visitor. Red flag: a conversion-only analysis with no unit of diversion defined.
How do you design allocation logic to minimize concurrent A/B test interactions?
Compare mutual exclusion with layered randomization via layers and reservations; stress isolation vs throughput. Designing allocation to isolate concurrent experiments.
Log Exposure, Not Just Assignment
Exposure logging records when a user actually sees a variant, not just assignment. Counting assigned but unexposed users dilutes treatment effects and biases lift. The common footgun is logging assignment as exposure, which silently underpowers your test.
How do network effects violate A/B tests and how to mitigate them?
Tests SUTVA violations and network experiment design. Answers note treated users alter control outcomes, then propose social-graph cluster randomization to isolate spillovers. Red flag: ignoring peer-to-peer spillover and using user-level randomization.
Why not stop an A/B test when it looks significant early?
Tests whether you understand repeated looks inflate false positives. The term is peeking: checking daily can turn a 5% Type I error rate into roughly 15% by day 3. Red flag: citing "low sample size" without stating that early stopping invalidates the p-value.
P-value vs confidence interval in an A/B test
A p-value gauges evidence against the null; a 95% CI shows plausible effect sizes and precision. Frequentist reasoning beyond binary significance. Calling the CI a 95% probability the true difference is inside.
What are your null and alternative hypotheses for this A/B test?
This tests translating a directional business question into statistical hypotheses. A strong answer states H0 as no difference in registration rate and H1 as green outperforming blue. A red flag is framing H0 as "blue is better" or using a two-tailed test.
How would you recommend launching a checkout flow with mixed A/B metrics?
This tests multi-metric trade-offs. A strong answer tags conversion as success and AOV as a guardrail, estimates net revenue impact, and frames decision as a risk-managed business choice. A red flag is demanding all metrics win or ignoring business context.
How does A/B testing native mobile apps differ from web?
Native mobile constraints versus web deployment. A strong answer covers: app store cycles forcing remote flags; device fragmentation skewing segments; and offline usage delaying events. Assuming mobile mirrors web with instant updates.
What are the engineering challenges and biases in overlapping A/B tests?
This tests inference across concurrent experiments. A strong answer names interaction effects and collision bias, proposes mutual exclusion or MVT for coupled UI, and stratified bucketing. Red flag: assuming additive lifts without combination validation.
Design an A/B test system for homepage headlines without deployments
Tests decoupling experiment configuration from deploys via a headless CMS and feature flagging. A strong answer covers variation containers, server-side bucketing, conversion tracking, and marketer-controlled copy swaps.
How would you track which headline wins in an A/B test?
Tests end-to-end experiment instrumentation across the stack. Outline: deterministically bucket users, serve variant A or B, emit click events, and aggregate by variant.
How do you diagnose a confounded A/B test with slower page load?
This tests confounding beyond p-values. A strong answer isolates the bug, checks if slower load hurt or helped conversions, and judges whether copy or latency caused the lift. A red flag is defending a 5% win because p < 0.05 while ignoring randomization.
Compare A/B/n testing with multi-armed bandits for headline optimization
Contrast A/B/n with dynamic MAB allocation; A/B/n optimizes final inference, MABs optimize reward; flag real-time infra. Grasp of explore-exploit and when rigor beats speed. Calling MABs always better ignoring regret.
How does cookie clearing affect A/B results and consistency?
This tests bucketing integrity when state changes re-randomize users. Explain that re-bucketing contaminates the sample and triggers SRM; propose deterministic server-side assignment, fingerprinting, or authenticated IDs.
Explain statistical significance in copy A/B tests and why one day fails.
This checks if you distinguish signal from noise. A strong answer defines statistical significance as confidence a difference is real, warns that one-day samples are small and skewed by variance, and cites false positive risk.
What CTA metric wins an A/B test and how to log it?
This tests connecting instrumentation to business outcomes via a click metric and tracked event. An answer picks click or conversion rate, fires an event with variant ID, and notes uniqueness. A red flag is using views without linking the event to the button.
How do you implement a CTA A/B test and attribute conversions?
This tests experiment architecture from bucketing to attribution. A strong answer covers: stable user bucketing, server or client-side rendering, and conversion events tagged with experiment and variant IDs.
Architect email subject line A/B testing for a large user base
Tests distributed systems and statistical rigor. Covers deterministic user bucketing, isolated variant delivery, deduplicated tracking, and pre-powered significance. Red flag: daily re-randomization or using open rate without confidence intervals.
Design a system to A/B test headlines for a single article URL
Hash users for sticky variants; store separately; emit events; compute CTR. controlled experiment design with user bucketing and attribution. client-side randomization without stickiness or event tracking.
Get AB Testing bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.