AB Testing
85 bites tagged AB Testing — interview questions with model answers, and 60-second explainers.
Instrumenting a feature to validate a qualitative insight
Restate the pain as a hypothesis, define a primary behavioral metric plus guardrails, instrument events, compare against baseline. Turning qualitative pain into measurable signals.
Checking if a 5% A/B lift is significant
State hypotheses, check predetermined sample and power, run the right test for the metric, read p-value and confidence interval. Hypothesis-testing fundamentals.
Designing an in-house A/B testing platform
Config service, deterministic bucketing SDK, metrics pipeline, analysis engine; fix peeking with fixed samples or sequential tests. System design plus statistical rigor.
Consistent A/B bucketing across sessions and devices
Hash a stable user ID with experiment salt, not random or cookie-only; beware caches keyed without variant. Deterministic, identity-based bucketing.
Client-side versus server-side A/B testing
Client-side flickers and is bypassable; server-side assigns before response, hides logic, suits backend changes. Where variant assignment and rendering happen.
Why A/B-only, no upfront research, costs engineers more
A/B testing optimizes within a chosen direction but cannot reveal the right problem; building wrong wastes engineering cycles and tech debt. defending upfront research on engineering-cost grounds.
Confidence intervals versus p-values, explained simply
Define a CI as a plausible range for the true effect with its precision, contrast it with a binary p-value, and read [-1%, 7%] as inconclusive because it spans zero. statistical literacy for A/B decisions.
Diagnose and validate a failing usability task
Pinpoint the failure step from session and telemetry data, form a root-cause hypothesis, ship a targeted fix, and re-test to validate. turning a usability failure into a rigorous engineering loop.
Present a small but significant A/B test lift
Hypothesis, design and validity checks, result with effect size and interval, business impact of 0.5%, then a clear recommendation. structuring an experiment narrative.
A/B test two fraud models in production
Randomize by entity, consider shadow mode first, collect precision/recall and business loss, decide with significance and guardrails. production model experimentation design.
Design an automated A/B test reporting system
Standardized metric definitions, automated stats with confidence intervals and guardrails, segment breakdowns, a clear ship recommendation. scalable experiment reporting design.
Primary metric up, guardrail down: ship or not?
Tie metrics to business value, weigh short-term lift against retention damage, use guardrails and an overall evaluation criterion. handling metric trade-offs.
Analyzing skewed revenue-per-user experiments
Heavy tails inflate variance and slow significance, and the mean is dominated by whales; mitigate via winsorization or capping, log transforms, CUPED variance reduction, or bootstrap and rank tests. handling skewed metrics in A/B tests.
Framing ad-load tradeoffs: revenue versus retention
Define revenue plus guardrail engagement metrics, run a long-enough experiment to see retention effects, and weigh short-term lift against lifetime-value erosion. balancing competing metrics over time.
Does forcing profile completion cause retention?
Name the confounder (engaged users self-select into completing profiles), warn that forcing it may not transfer the effect, and propose a randomized experiment. distinguishing correlation from causation.
Multi-armed bandit vs A/B testing for headlines
Bandits shift traffic toward winners using updated per-arm reward state; A/B holds fixed splits until a fixed-horizon decision. bandits versus A/B and explore/exploit. calling a bandit just an A/B test that ends early.
Manage localized copy across many A/B tests
A keyed versioned string store, a translation workflow with per-locale status, and pooling or hierarchical analysis for small languages. localization plus experimentation at scale.
Build a simple A/B test for a headline
Assign each visitor a sticky bucket, serve the matching headline variant, log impressions and conversions per variant. basic A/B test mechanics. re-randomizing on every load so a user sees both variants.
Determine A/B test sample size
Define baseline rate, minimum detectable effect, significance (alpha), and power (1-beta); smaller effects and stricter thresholds need more users. the inputs to a power calculation. ignoring power or treating MDE as fixed.
Interpret a p-value in an A/B test
Define it as P(data this extreme | null true), interpret 0.03 against a 0.05 threshold, and state what it is NOT. correct meaning of a p-value. claiming a 3% chance the result is wrong or that the effect is.
Implementing a consistent-assignment A/B test
Need an assignment service, exposure logging, and event tracking; ensure stickiness by hashing a stable user id; analyze conversion per variant. End-to-end experiment engineering. Bucketing on session or device.
The multiple comparisons problem in A/B testing
Many tests at alpha 0.05 inflate the chance of a false positive; mitigate with Bonferroni or FDR control plus pre-registered metrics. Statistical rigor at scale. Cherry-picking whichever metric crosses p<0.05.
Designing a self-serve experimentation platform
SDK with sane defaults, automated pre-launch validation, sample-ratio and guardrail-metric checks. building safe experimentation as a platform, not a service.
Peeking in A/B tests and how to mitigate it
Peeking is checking significance repeatedly and stopping at the first significant result, which inflates false positives; mitigate with fixed sample sizes or sequential… understanding inflated false positives from repeated looks.
Get AB Testing bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.