AB Testing
85 bites tagged AB Testing — interview questions with model answers, and 60-second explainers.
Implement a welcome-message A/B test
Deterministic hash of a stable ID for sticky assignment, conditional rendering of the personalized variant, and exposure plus click tracking keyed to the same ID. end-to-end client experiment basics.
Communicate experiment results and check guardrails
Explain the lift and confidence in plain business terms, report a confidence interval not just a point, and verify guardrails before shipping. translating statistics and protecting against harm.
Components of a testable A/B hypothesis
A specific change, a predicted directional effect on one primary metric, a rationale, and a measurable success threshold. experimental rigor before launch.
Design a streak-saver mechanism
Timezone-aware streak state, capped saver inventory with replenish rules, and A/B testing forgiveness against retention plus guardrails. modeling stateful engagement with behavioral nuance.
Experiment design under network effects
Cluster-level randomization, graph or geo clustering to contain spillover, and analysis at the cluster unit. awareness that SUTVA breaks under interference.
Set up a client-side button color A/B test
Stable hashing of a persistent ID into buckets, conditional rendering of the variant, exposure plus click event logging. practical experiment wiring on the client. re-randomizing on each load so users flicker between variants.
Trace an event from click to analysis
Client SDK captures and batches, a collection endpoint ingests, a stream and ETL enrich and load into a warehouse for analysis. end-to-end understanding of an analytics event pipeline.
When user-level A/B tests get contaminated
Network or marketplace spillover violates SUTVA, so randomize by cluster (geo, group, time) and analyze at that level. recognizing interference that breaks the independence assumption.
Explain statistical power and respond to extending a null A/B test
This tests statistical power and p-hacking judgment. A strong answer defines power as detecting a true effect, rejects extending the test to chase significance, and requires pre-registered sample sizes. Agreeing to run until it hits significance is a red flag.
Explain what a p-value represents in an A/B test and interpret 0.03
This tests frequentist hypothesis testing literacy. It is the chance of seeing a result this extreme if the null of no effect were true; 0.03 means 3% chance of such data under the null, not 97% chance the variant wins. Red flag: saying 97% likely.
T-test: Are Two Groups Actually Different?
A t-test asks whether the gap between two groups is a real signal or just sampling noise. UX researchers use it to compare task times or conversion rates between designs. The footgun is trusting results from tiny samples where the math gives false confidence.
Implement A/B testing with Middleware rewrites and cookies
Sticky cookie, internal rewrite, vary cache on cookie, server-side analytics. Using Next.js Middleware to split traffic and the cache or analytics impact. Client redirects or ignoring cache collisions.
How do you technically implement an A/B test for onboarding flows?
Tests experiment pipeline design: deterministic user bucketing, an exposure event before rendering, and an event schema linking actions to variant_id. Red flag: re-randomizing per session or skipping exposure logs.
Design a follow-up experiment to resolve conflicting qualitative and quantitative data
This tests mixed-methods integration. Strong answers sequence experiments: observe users in the low-engagement flow for friction, then run a higher-fidelity quantitative test with behavioral metrics tied to stated intent.
How would you design an A/B test for two live ML models?
Tests production experimentation rigor beyond random splitting. Strong answers cover: consistent user hashing for sticky assignment, isolated feature stores, guardrail metrics, and pre-calculated statistical power.
Explain the difference between statistical and practical significance
Define statistical vs practical significance; note large samples make tiny effects significant; give a real example. If you know p-values detect real effects but ignore value. Using p < 0.05 alone to justify shipping.
Describe cluster or switchback randomization for network-effect A/B tests.
Tests SUTVA violation, cluster-switchback tradeoffs. Outline: cluster (geo, teams) or switchback (time slices) isolation; cover pipeline changes, 10x inflation, and correlated error. Red flag: user-level randomization with post-hoc fixes or ignoring spillover.
What experiment artifacts and metrics do you present to decide shipping?
This tests structured experiment discipline. A strong answer covers the pre-approved design doc, scorecard results for primary goal and guardrail metrics plus secondary breakdowns versus the hypothesis, and duration context.
How do you mitigate peeking in experiment infrastructure?
Lock results behind minimum samples; auto-correct via sequential testing; hide early metrics and require stop approval. infrastructure enforcing valid reads through gates and incentives. relying on training not systems.
Design an assignment service for an experimentation platform
This tests deterministic hashing and independence in randomization. A strong answer lists user ID, experiment salt, and allocations; uses salted hash for consistency; and warns that separate exposure and bucket hashes with mod can cause sample ratio mismatch.
How do you ensure consistent A/B test variants across sessions and devices?
Tests stable identity and delayed attribution. Fix: one stable user ID for SDK bucketing, persisted across devices via login or first-party cookies, attached to every conversion event. Never use per-device randomization or third-party cookies.
Design an A/B test for a Buy Now button
This tests experiment plumbing: deterministic bucketing, sticky storage, and logging. A strong answer covers user-ID hashing, cookie persistence, and impression-plus-conversion events.
Long-term onboarding holdback: technical and data integrity challenges
This tests the engineering cost of year-long holdbacks in growth. A strong answer covers feature-flag entropy, pipeline drift, survivorship bias, and counterfactual validity. Red flag: treating the holdback as static config that never rots.
Architect an A/B test for paid-ad signup flows
Tests pre-auth bucketing and funnel attribution. Hash a stable anonymous ID for fast assignment; stream events via Kafka into hourly aggregates; run t-tests on signup rates. Red flag: assigning after signup starts or DB lookups per assignment.
Get AB Testing bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.