AB Testing
85 bites tagged AB Testing: interview questions with model answers, and 60-second explainers.
Test yourself: Top 30 AB Testing interview questions →Multiple choice, with the correct answer and why it is correct on every question. Free, no sign-in.Instrumenting a feature to validate a qualitative insight
Restate the pain as a hypothesis, define a primary behavioral metric plus guardrails, instrument events, compare against baseline.
Checking if a 5% A/B lift is significant
State hypotheses, check predetermined sample and power, run the right test for the metric, read p-value and confidence interval.
Designing an in-house A/B testing platform
Config service, deterministic bucketing SDK, metrics pipeline, analysis engine; fix peeking with fixed samples or sequential tests.
Consistent A/B bucketing across sessions and devices
Hash a stable user ID with experiment salt, not random or cookie-only; beware caches keyed without variant.
Client-side versus server-side A/B testing
Client-side flickers and is bypassable; server-side assigns before response, hides logic, suits backend changes.
Why A/B-only, no upfront research, costs engineers more
A/B testing optimizes within a chosen direction but cannot reveal the right problem; building wrong wastes engineering cycles and tech debt.
Confidence intervals versus p-values, explained simply
Define a CI as a plausible range for the true effect with its precision, contrast it with a binary p-value, and read [-1%, 7%] as inconclusive because it spans zero.
Diagnose and validate a failing usability task
Pinpoint the failure step from session and telemetry data, form a root-cause hypothesis, ship a targeted fix, and re-test to validate.
Present a small but significant A/B test lift
Hypothesis, design and validity checks, result with effect size and interval, business impact of 0.5%, then a clear recommendation.
A/B test two fraud models in production
Randomize by entity, consider shadow mode first, collect precision/recall and business loss, decide with significance and guardrails.
Design an automated A/B test reporting system
Standardized metric definitions, automated stats with confidence intervals and guardrails, segment breakdowns, a clear ship recommendation.
Primary metric up, guardrail down: ship or not?
Tie metrics to business value, weigh short-term lift against retention damage, use guardrails and an overall evaluation criterion.
Analyzing skewed revenue-per-user experiments
Heavy tails inflate variance and slow significance, and the mean is dominated by whales; mitigate via winsorization or capping, log transforms, CUPED variance reduction, or bootstrap and rank tests.
Framing ad-load tradeoffs: revenue versus retention
Define revenue plus guardrail engagement metrics, run a long-enough experiment to see retention effects, and weigh short-term lift against lifetime-value erosion.
Does forcing profile completion cause retention?
Name the confounder (engaged users self-select into completing profiles), warn that forcing it may not transfer the effect, and propose a randomized experiment.
Multi-armed bandit vs A/B testing for headlines
Bandits shift traffic toward winners using updated per-arm reward state; A/B holds fixed splits until a fixed-horizon decision.
Manage localized copy across many A/B tests
A keyed versioned string store, a translation workflow with per-locale status, and pooling or hierarchical analysis for small languages.
Build a simple A/B test for a headline
Assign each visitor a sticky bucket, serve the matching headline variant, log impressions and conversions per variant.
Determine A/B test sample size
Define baseline rate, minimum detectable effect, significance (alpha), and power (1-beta); smaller effects and stricter thresholds need more users.
Interpret a p-value in an A/B test
Define it as P(data this extreme | null true), interpret 0.03 against a 0.05 threshold, and state what it is NOT.
Implementing a consistent-assignment A/B test
Need an assignment service, exposure logging, and event tracking; ensure stickiness by hashing a stable user id; analyze conversion per variant.
The multiple comparisons problem in A/B testing
Many tests at alpha 0.05 inflate the chance of a false positive; mitigate with Bonferroni or FDR control plus pre-registered metrics.
Designing a self-serve experimentation platform
SDK with sane defaults, automated pre-launch validation, sample-ratio and guardrail-metric checks.
Peeking in A/B tests and how to mitigate it
Peeking is checking significance repeatedly and stopping at the first significant result, which inflates false positives; mitigate with fixed sample sizes or sequential…
Get AB Testing bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.