AB Testing
85 bites tagged AB Testing — interview questions with model answers, and 60-second explainers.
Architect a real-time multi-armed bandit and compare trade-offs to A/B testing
Sketch a fast arm router, streaming feedback, and model updates; contrast MAB regret minimization with A/B's unbiased estimates. Real-time ML serving and statistical trade-offs.
A/B test copy in a native app without app store releases
Tests native release friction versus web. Strong answers cite app store review, binary caching, and stale config; then remote fetch, local defaults, and user targeting. A red flag is assuming updates are instant like the web.
Describe how you'd implement an A/B test for a landing page headline
Mention deterministic bucketing, anti-flicker rendering, tracking with variant IDs, and sample-size planning. Experiment integrity beyond DOM swaps. Swapping headlines client-side after paint biases data and hurts UX.
How would you A/B test sign-up button copy and measure results?
This tests basic experimental design. A strong answer covers: random assignment, serving variant copy, tracking impressions and conversions, and measuring lift. A red flag is sequential testing or vanity metrics like clicks without sign-ups.
Explain how CUPED increases statistical power and required data
Tests ANCOVA variance reduction. Answer: CUPED regresses pre-experiment X on Y, shrinking variance by (1-ρ²); needs pre-randomization prognostic baseline; beats difference scores. Red flag: calling it Y-X subtraction or saying it changes the effect.
What is the 'novelty effect' in experimentation?
Tests whether you separate temporary curiosity from durable value. A strong answer defines novelty effect as short-term behavior change triggered by new elements, notes it inflates early experiment lift, and proposes longer runtimes or lagged cohort analysis.
Why can't you t-test p99 latency, and what's a valid alternative?
Explain that t-tests target means while p99 variance depends on tail density; propose bootstrap CIs or permutation tests. Knowing percentiles are order statistics, not means. Invoking CLT to justify a t-test on p99.
A/B test p-value 0.08, PM wants to ship. How do you advise?
Tests statistical rigor versus business pragmatism. A strong answer covers pre-registered thresholds, false positive risk, statistical power, confidence intervals, and the business cost of being wrong. Red flag: shipping without quantifying downside risk.
What does a p-value of 0.03 mean at alpha 0.05?
0.03 means 3% chance of data this extreme if the null (no effect) holds; since 0.03 < 0.05, reject the null. Literal p-value interpretation. Calling it the probability the null is true or the effect is 97% real.
A/B test shows 5% lift with imbalanced mobile traffic. What next?
This tests sample ratio mismatch and Simpson's paradox. A strong answer says the lift is confounded by platform, demands stratified analysis to isolate the true effect, and checks randomization. A red flag is accepting the 5% lift without questioning skew.
Pitfalls of using conversion rate as a checkout North Star?
Tests if you know over-optimizing conversion can degrade revenue quality or trust. Strong answers cite lower AOV or fraud risks, then list guardrails like refund rate, lifetime value, and checkout errors. Red flag: insisting conversion is the sole metric.
How would you implement an A/B test for a redesigned dashboard?
Tests experiment rigor: randomize stable user buckets, pick one primary metric and guardrails like latency, pre-calculate sample size and runtime with a minimum detectable effect, and validate via backend logging.
Pitfalls of 'Conversion Rate' as a North Star Metric
This tests your ability to see beyond a single metric and understand its second-order effects. A strong answer identifies pitfalls like lower AOV, then proposes counter-metrics (AOV, return rate) and guardrail metrics (page load time).
Get AB Testing bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.