Top 30 AB Testing Interview Questions and Answers

30 questions. Pick an answer, or open “Show the answer” to read it.

Answers are graded in your browser. Nothing is saved, and no XP or streak is earned here. The app keeps score.

1. Question 1 of 30

   A checkout redesign raises conversion but sharply increases chargebacks and refund requests. What does this illustrate about using conversion as a North Star?

   • aIsolating a single metric can obscure degradations in revenue quality and trust
   • bGuardrails are primarily needed to validate the statistical significance of tests
   • cHigher conversion always correlates with healthier long-term revenue growth
   • dConversion rate should be replaced by lifetime value as the only success metric
   Show the answer

   Answer: a · Isolating a single metric can obscure degradations in revenue quality and trust

   The scenario exemplifies the core pitfall that optimizing conversion alone ignores the profitability and quality of converted traffic, which guardrail metrics are designed to surface. Option C echoes the dangerous misconception that revenue automatically follows conversion, while Option B confuses business guardrails with experimental validity checks.

   Read the full bite: Pitfalls of using conversion rate as a checkout North Star?

2. Question 2 of 30

   A product team successfully increases conversion rate by simplifying a user flow. What is the MOST critical next step to validate that this change is a genuine improvement for the business?

   • aAnalyze user engagement metrics like time on page and bounce rate to ensure the simplified flow hasn't negatively impacted user interaction.
   • bMonitor the new conversion rate for a longer period to confirm its stability and statistical significance.
   • cExamine counter-metrics such as Average Order Value and return rates, and guardrail metrics like page load time and customer support tickets.
   • dConduct qualitative user research, like interviews or usability tests, to understand user perception of the new simplified flow.
   Show the answer

   Answer: c · Examine counter-metrics such as Average Order Value and return rates, and guardrail metrics like page load time and customer support tickets.

   The card emphasizes that an increased conversion rate can mask negative impacts on other crucial business metrics (like Average Order Value, return rate) and system health (like page load time, support tickets). Therefore, examining these counter-metrics and guardrail metrics is critical to determine if the change is a true success, rather than a local optimization causing global problems.

   Read the full bite: Pitfalls of 'Conversion Rate' as a North Star Metric

3. Question 3 of 30

   A checkout A/B test shows significantly higher conversion but slightly lower AOV. What is the strongest basis for a launch recommendation?

   • aLaunch because conversion is up and automatically outweighs any decrease in basket size
   • bRequire AOV to also achieve statistical significance as a co-success metric
   • cAdjust p-values for multiple comparisons and launch only if both metrics remain significant
   • dEstimate net revenue and ensure the AOV decline is within a pre-specified non-inferiority margin
   Show the answer

   Answer: d · Estimate net revenue and ensure the AOV decline is within a pre-specified non-inferiority margin

   The right framework classifies conversion as a success metric and AOV as a guardrail requiring non-inferiority, then uses net revenue to judge the business outcome. Treating AOV as a co-success metric that must significantly increase is a tempting error that causes unnecessary conservatism and missed wins.

   Read the full bite: How would you recommend launching a checkout flow with mixed A/B metrics?

4. Question 4 of 30

   Which approach best minimizes interaction bias across concurrent experiments while preserving platform velocity?

   • aApply mutual exclusion globally so each user sees only one experiment at a time
   • bReduce traffic percentages for each experiment so overlaps are uncommon
   • cAssign users randomly to all experiments and add interaction terms during analysis
   • dUse orthogonal layers with reservation amounts, restricting mutual exclusion to tightly coupled features
   Show the answer

   Answer: d · Use orthogonal layers with reservation amounts, restricting mutual exclusion to tightly coupled features

   Orthogonal layers isolate independent experiments via separate randomization units and reservation amounts prevent layer starvation, while mutual exclusion is reserved for high-risk features because global use destroys velocity. The most tempting distractor, adding post-hoc interaction terms, fails because clean causal inference requires designed allocation—regression cannot fix unstructured overlaps after the fact.

   Read the full bite: How do you design allocation logic to minimize concurrent A/B test interactions?

5. Question 5 of 30

   When A/B testing sign-up button copy, which practice ensures you can accurately attribute differences in sign-ups to the copy change?

   • aRandomly assign users to variants using a hashed user ID, track impressions and conversions, and compare conversion rates
   • bShow version A for one week and version B for the next to avoid overlap
   • cSplit users by geolocation and compare sign-up counts across regions after two weeks
   • dTrack button clicks for each variant and compare click counts, assuming more clicks lead to more sign-ups
   Show the answer

   Answer: a · Randomly assign users to variants using a hashed user ID, track impressions and conversions, and compare conversion rates

   Randomizing with hashed user IDs and tracking both impressions and completed sign-ups lets you calculate a true conversion rate and establish causality. Sequential testing introduces temporal bias because traffic composition and external factors differ between weeks, invalidating the comparison.

   Read the full bite: How would you A/B test sign-up button copy and measure results?

6. Question 6 of 30

   Why is the correlation between profile completion and retention weak evidence that forcing completion will improve retention?

   • aCorrelation coefficients cannot apply to user behavior data
   • bRetention can never be measured accurately for new users
   • cThe sample of completed profiles is always too small to trust
   • dEngaged users self-select into completing profiles, confounding the relationship
   Show the answer

   Answer: d · Engaged users self-select into completing profiles, confounding the relationship

   Motivated users both complete profiles and retain, so engagement is a confounder making completion a marker rather than a proven cause. A randomized experiment is needed; the other options misstate the actual problem.

   Read the full bite: Does forcing profile completion cause retention?

7. Question 7 of 30

   What is the most important addition to a revenue metric when evaluating a change that may harm long-term satisfaction?

   • aMeasuring revenue more precisely with a larger sample
   • bLong-horizon guardrail metrics like retention plus a sufficiently long experiment
   • cShipping quickly to capture the revenue before users notice
   • dRelying on stakeholder intuition about user happiness
   Show the answer

   Answer: b · Long-horizon guardrail metrics like retention plus a sufficiently long experiment

   Long-term harm surfaces as delayed churn, so guardrail metrics measured over a long-enough window are essential to weigh against the immediate revenue lift. More precise short-term revenue alone still misses the delayed retention cost.

   Read the full bite: Framing ad-load tradeoffs: revenue versus retention

8. Question 8 of 30

   Which hypothesis pair best tests whether a green signup button increases registrations compared to blue?

   • aH0: There is no difference in registration conversion rate; H1: The registration conversion rates are not equal.
   • bH0: The registration conversion rate for green equals that for blue; H1: The registration conversion rate for green is strictly greater than that for blue.
   • cH0: The blue button has a higher registration conversion rate than the green button; H1: The green button has a higher registration conversion rate than the blue button.
   • dH0: The green and blue buttons perform the same; H1: The green button performs better.
   Show the answer

   Answer: b · H0: The registration conversion rate for green equals that for blue; H1: The registration conversion rate for green is strictly greater than that for blue.

   The correct answer frames H0 as no difference and H1 as a directional increase in the specific metric, matching the one-tailed nature of the business question. Option A is tempting because it uses 'no difference,' but it wrongly uses a two-tailed alternative that ignores the directional ask and wastes statistical power.

   Read the full bite: What are your null and alternative hypotheses for this A/B test?

9. Question 9 of 30

   Which sequencing strategy best resolves a conflict between qualitative intent data and low-engagement quantitative A/B results?

   • aGenerate hypotheses for the conflict, run a targeted qualitative study on the A/B flow, then redesign the quantitative experiment with better behavioral proxies and a decision gate
   • bMerge the interview transcripts and click-through rates into a single weighted score to identify the true user preference
   • cRun the qualitative and quantitative investigations in parallel without a shared research question, reconciling findings only in the final report
   • dIncrease A/B traffic and run a survey as a tiebreaker to resolve the discrepancy
   Show the answer

   Answer: a · Generate hypotheses for the conflict, run a targeted qualitative study on the A/B flow, then redesign the quantitative experiment with better behavioral proxies and a decision gate

   This approach correctly sequences hypothesis generation, targeted qualitative friction-finding on the exact A/B flow, and a redesigned higher-fidelity quantitative experiment with a unified decision gate. Option D is tempting because increasing sample size feels rigorous, but rerunning the same test and using a survey fails to diagnose the root cause of the say-do gap and relies on a method that explains neither behavior nor motivation deeply.

   Read the full bite: Design a follow-up experiment to resolve conflicting qualitative and quantitative data

10. Question 10 of 30

    To isolate the effect of loss-aversion versus gain framing at checkout, which assignment approach ensures the same shopper always sees the same variant and avoids within-subject contamination?

    • aAssign using a hash of the stable user ID before the checkout page renders
    • bSplit traffic by even and odd calendar days for the entire population
    • cRandomize by session so returning users may see either variant
    • dRotate the banner by time of day to control for traffic patterns
    Show the answer

    Answer: a · Assign using a hash of the stable user ID before the checkout page renders

    The card requires a stable user-level unit of diversion and pre-allocation before rendering to prevent contamination and flicker bias. Session-level randomization is explicitly flagged as a red flag because it splits users across variants and invalidates the experiment.

    Read the full bite: Design an A/B test for loss aversion versus gain framing at checkout

11. Question 11 of 30

    When running an experiment to increase CTA clicks, which best describes the proper role of a guardrail metric like revenue per user?

    • aIt should be evaluated with a two-sided test to detect any change in business health.
    • bIt serves as a secondary success metric reviewed only after the primary metric improves.
    • cIt requires a pre-set non-inferiority threshold and halts the experiment if breached, regardless of CTA lift.
    • dIt can be skipped if the primary metric shows a large enough positive effect.
    Show the answer

    Answer: c · It requires a pre-set non-inferiority threshold and halts the experiment if breached, regardless of CTA lift.

    Guardrails are hard safety limits with pre-set non-inferiority thresholds that trigger an experiment stop if breached, even when the primary metric wins. Treating them as secondary success metrics reviewed only after the primary metric succeeds is a common misconception that risks silently damaging the business.

    Read the full bite: When increasing CTA clicks, what side-effects and guardrails should you consider?

12. Question 12 of 30

    You need to deploy a headline A/B test on a landing page. Which setup best balances statistical validity, attribution accuracy, and user experience?

    • aHash a user ID or cookie to lock the user into one variant across sessions, render the headline server-side or before paint, and include the variant ID in exposure and conversion events.
    • bRandomly assign a variant with a client-side script that runs after the DOM loads, swapping the headline text on each visit.
    • cRun the test until one variant reaches 100 sign-ups and declare it the winner based on the higher raw count.
    • dServe the headline via server-side rendering but re-bucket the user on every page refresh to increase the sample size.
    Show the answer

    Answer: a · Hash a user ID or cookie to lock the user into one variant across sessions, render the headline server-side or before paint, and include the variant ID in exposure and conversion events.

    Deterministic bucketing and server-side rendering prevent flicker and ensure consistent attribution, while tagged events let analysts join exposures to conversions. Option B is wrong because client-side swaps after paint create layout shift and users may convert before the variant is applied, biasing results.

    Read the full bite: Describe how you'd implement an A/B test for a landing page headline

13. Question 13 of 30

    When setting up a button color A/B test, what separates a valid experiment from a casual tweak?

    • aPre-registering the hypothesis, primary metric, sample size, and decision criteria before launch
    • bRunning the test for a set period because it feels long enough and reviewing all metrics afterward
    • cComparing this week's green button to last week's blue button to measure lift
    • dRandomizing visitors by device type to ensure an even split across variants
    Show the answer

    Answer: a · Pre-registering the hypothesis, primary metric, sample size, and decision criteria before launch

    Pre-registering the hypothesis, primary metric, sample size, and decision criteria establishes the structured framework that defines a valid experiment. Comparing this week against last week is a tempting shortcut but introduces seasonality bias by omitting a concurrent control group.

    Read the full bite: How do you set up a button color A/B test?

14. Question 14 of 30

    In an A/B test, you observe a conversion lift with a p-value of 0.02. Which statement correctly interprets this p-value?

    • aThere is a 2% chance that the test conclusion is wrong and the null hypothesis is true.
    • bIf the variant actually has no effect, there is a 2% chance of observing a lift this large or larger purely due to random variation.
    • cThere is a 98% probability that the variant is genuinely better than the control.
    • dThe false discovery rate is 2%, meaning only 2% of significant results like this one are false positives.
    Show the answer

    Answer: b · If the variant actually has no effect, there is a 2% chance of observing a lift this large or larger purely due to random variation.

    A p-value assumes the null hypothesis is true and measures the probability of seeing a result at least this extreme under that assumption; it is not the probability that the variant is better, which is the most common misinterpretation.

    Read the full bite: What is a p-value in A/B tests and what does threshold mean?

15. Question 15 of 30

    When designing an A/B test for copy in a native app without app store releases, what is the primary architectural pattern that safely decouples content changes from binary updates?

    • aBundling all experiment variants into the app binary at build time to guarantee offline access
    • bFetching experiment parameters asynchronously via remote config while merging them with local defaults
    • cRequiring users to force-update the app before enrolling in any new experiment
    • dUsing a CMS to update copy in real time just like on a website, since modern apps render content dynamically
    Show the answer

    Answer: b · Fetching experiment parameters asynchronously via remote config while merging them with local defaults

    Remote config merged with local defaults lets the app render immediately and change copy without a store release. Treating native apps like websites and using a CMS is wrong because binaries bundle assets at build time and cannot update instantly like the web.

    Read the full bite: A/B test copy in a native app without app store releases

16. Question 16 of 30

    An experiment needs 60,000 users per variant to detect a 5% lift with 80% power. Why does detecting a 2% lift at the same power require 370,000 users?

    • aBecause the significance level must be tightened from 5% to 1% for finer detection
    • bBecause the test must distinguish a weaker signal from the same random noise
    • cBecause the smaller lift reduces the test's statistical significance, requiring more users to restore it
    • dBecause maintaining 80% power at a smaller MDE raises the false positive rate
    Show the answer

    Answer: b · Because the test must distinguish a weaker signal from the same random noise

    The card explains that shrinking the MDE inflates sample size because you are asking the test to resolve a smaller signal from the same noise. Distractor C reflects the common misconception of conflating power/MDE with the false positive rate (alpha), which is held constant.

    Read the full bite: Explain statistical power, MDE, and sample size impact

17. Question 17 of 30

    In the discount code example, users never applied a promo code because the field was collapsed by default. What kind of usability failure was this, and how was the diagnosis confirmed?

    • aA comprehension failure, confirmed because users clicked the field but typed invalid codes.
    • bA discoverability failure, confirmed by session recordings showing users never found the collapsed field, corroborated by near-zero clicks in telemetry.
    • cAn interaction failure, confirmed because users clicked the field repeatedly without success.
    • dA feedback failure, confirmed because users applied the code but could not tell whether it worked.
    Show the answer

    Answer: b · A discoverability failure, confirmed by session recordings showing users never found the collapsed field, corroborated by near-zero clicks in telemetry.

    The card labels this a discoverability failure because users never located the collapsed control at all, confirmed by both recordings and near-zero click telemetry. An interaction failure (option C) would mean they found the control but struggled to operate it, which is not what happened.

    Read the full bite: Diagnose and validate a failing usability task

18. Question 18 of 30

    You switch from user-level to city-level randomization to avoid spillover. What must also change to keep results valid?

    • aYou should drop the control group entirely
    • bThe analysis must treat the city, not the user, as the unit, lowering effective power
    • cNothing; you can still compute variance per individual user
    • dYou must increase the per-user sample size to keep power identical
    Show the answer

    Answer: b · The analysis must treat the city, not the user, as the unit, lowering effective power

    Clustered assignment means users within a cluster are correlated, so variance must be estimated across clusters, which reduces effective sample size and power. Analyzing per user understates variance and fabricates significance.

    Read the full bite: When user-level A/B tests get contaminated

19. Question 19 of 30

    What makes standard post-hoc hypothesis testing invalid after running a multi-armed bandit campaign?

    • aBayesian updating and frequentist t-tests rely on fundamentally different definitions of probability
    • bSub-100 ms latency requirements limit the total observations available for classical inference
    • cAdaptive allocation shifts traffic toward leading arms, biasing sample sizes and violating fixed-sample assumptions
    • dThe exploration floor fragments traffic so that no single arm receives a large enough fixed sample
    Show the answer

    Answer: c · Adaptive allocation shifts traffic toward leading arms, biasing sample sizes and violating fixed-sample assumptions

    The card states that MAB's adaptive traffic allocation corrupts the fixed-sample assumptions required for classical hypothesis testing, producing biased lift estimates. Option D is tempting because it mentions fixed samples, but the exploration floor is an operational guardrail, not the source of the statistical bias.

    Read the full bite: Architect a real-time multi-armed bandit and compare trade-offs to A/B testing

20. Question 20 of 30

    When bucketing anonymous paid-ad traffic for a signup-flow A/B test, which approach best satisfies speed, scale, and isolation requirements?

    • aUse a centralized Redis cache to store per-user variant assignments, retrieved on every page load to guarantee consistency across devices.
    • bAssign after the user submits the first signup form, then store the variant in a user database row to ensure authenticated attribution.
    • cCompute a deterministic hash of the anonymous ID salted with the experiment ID at the landing page, store the variant in a cookie, and skip database lookups entirely.
    • dApply modulo 100 directly to the raw user identifier without salting, assigning the same bucket to the same user across all experiments.
    Show the answer

    Answer: c · Compute a deterministic hash of the anonymous ID salted with the experiment ID at the landing page, store the variant in a cookie, and skip database lookups entirely.

    Deterministic hashing with experiment salting at the landing page eliminates database latency and prevents cross-experiment correlation, while persisting the variant in a cookie maintains session consistency. Option B is tempting because database storage feels reliable, but delaying assignment until after signup starts severs attribution from the paid ad click and invalidates the experiment.

    Read the full bite: Architect an A/B test for paid-ad signup flows

21. Question 21 of 30

    When an A/B test shows a 5% aggregate lift but treatment has 80% mobile users versus 50% in control, what is the most appropriate next step?

    • aWeight the aggregate metric by platform share but skip separate per-platform treatment effect estimates
    • bKeep the test running longer until the platform mix naturally equalizes between treatment and control
    • cStratify the analysis by platform and debug the assignment pipeline before trusting any lift estimate
    • dTrust the 5% lift because the total sample size is large enough to withstand the platform imbalance
    Show the answer

    Answer: c · Stratify the analysis by platform and debug the assignment pipeline before trusting any lift estimate

    Stratifying by platform isolates the true treatment effect from the confounding platform skew, and debugging the assignment pipeline identifies the root cause. The most tempting distractor—trusting the lift because the sample size is large—is wrong because sample ratio mismatch breaks randomization no matter how large N is.

    Read the full bite: A/B test shows 5% lift with imbalanced mobile traffic. What next?

22. Question 22 of 30

    Which design best supports a valid headline A/B test on a single article URL with reliable CTR measurement?

    • aCreate a separate article row for each headline variant and assign users via server-side randomization on every request
    • bUse client-side randomization that re-rolls on every page load, keep the article table unchanged, and emit impression and click events from the browser
    • cStore the variant headline directly on the article record and update real-time click and impression counters in place
    • dHash the user ID with the experiment key for deterministic bucketing, maintain separate experiment and variant tables, and emit structured impression and click events
    Show the answer

    Answer: d · Hash the user ID with the experiment key for deterministic bucketing, maintain separate experiment and variant tables, and emit structured impression and click events

    Deterministic hashing ensures the same user always sees the same headline across sessions without per-user assignment rows, separate tables preserve the single URL requirement, and discrete events enable accurate CTR and statistical testing. Option B is tempting because it tracks events and avoids schema changes, but re-rolling client-side splits users across variants and corrupts the experiment data.

    Read the full bite: Design a system to A/B test headlines for a single article URL

23. Question 23 of 30

    Six months into a year-long holdback, the team ships an AI assistant that logs new events the holdback cannot trigger. What is the most robust way to preserve metric comparability?

    • aRe-randomize users at month six to ensure both cohorts have equal probability of seeing the new assistant.
    • bBackfill synthetic zero values for the missing assistant events in the holdback group to maintain schema consistency.
    • cVersion the metric schema and compute engagement using only pre-assistant events for both groups.
    • dContinue comparing raw engagement scores between groups while treating the holdback as static config that does not need pipeline updates.
    Show the answer

    Answer: c · Version the metric schema and compute engagement using only pre-assistant events for both groups.

    Versioning the metric schema ensures both groups are evaluated on comparable event definitions, preserving causal validity. Backfilling synthetic zeros for the holdback distorts the engagement distribution and creates spurious differences between groups.

    Read the full bite: Long-term onboarding holdback: technical and data integrity challenges

24. Question 24 of 30

    Which statement correctly interprets a p-value of 0.03 when alpha is 0.05?

    • aThere is a 3% probability that the null hypothesis is true, meaning the feature has almost no chance of being ineffective.
    • bThe p-value indicates there is only a 3% chance the observed difference was caused by random noise alone.
    • cIf the null hypothesis is true, there is a 3% chance of seeing data at least this extreme, so we reject the null.
    • dBecause 0.03 is below 0.05, the experiment proves the feature will have a meaningful business impact.
    Show the answer

    Answer: c · If the null hypothesis is true, there is a 3% chance of seeing data at least this extreme, so we reject the null.

    Option C is correct because a p-value of 0.03 means there is a 3% probability of observing data this extreme if the null were true, and since that is below the pre-registered alpha of 0.05, we reject the null. Option A is tempting but wrong because it treats the p-value as the probability that the null hypothesis is true, which is a Bayesian interpretation and not valid under frequentist inference.

    Read the full bite: What does a p-value of 0.03 mean at alpha 0.05?

25. Question 25 of 30

    An A/B test returns p=0.08 and the PM wants to ship. What is the strongest recommendation?

    • aTreat the result as a 92% probability that the variant is better and approve the launch
    • bRefuse to ship and rerun the exact same test until the p-value drops below 0.05
    • cReview the confidence interval, quantify the business cost of error, and propose a larger test or monitored rollout
    • dShip immediately because the result is directionally positive and close to significant
    Show the answer

    Answer: c · Review the confidence interval, quantify the business cost of error, and propose a larger test or monitored rollout

    This response balances statistical rigor with business pragmatism by checking whether the plausible effect range includes harm and weighing the cost of error before acting. Shipping simply because 0.08 is near 0.05 abandons the pre-specified threshold and inflates the false positive rate through p-hacking.

    Read the full bite: A/B test p-value 0.08, PM wants to ship. How do you advise?

26. Question 26 of 30

    Why is deterministic bucketing preferred over client-side random assignment in a Buy Now button A/B test?

    • aIt prevents users from reshuffling variants across visits, preserving statistical independence
    • bIt allows the browser to cache the button CSS across both variants
    • cIt reduces the payload size of analytics events sent to the warehouse
    • dIt eliminates the need to log impression events before conversions
    Show the answer

    Answer: a · It prevents users from reshuffling variants across visits, preserving statistical independence

    Deterministic hashing guarantees the same user always lands in the same bucket, whereas Math.random reshuffles users on every visit and breaks statistical independence. Option D is tempting because candidates often want to simplify telemetry, but omitting impression events makes it impossible to compute click-through rates or detect sample ratio mismatch.

    Read the full bite: Design an A/B test for a Buy Now button

27. Question 27 of 30

    Why is a two-sample t-test fundamentally unsuitable for comparing p99 latency between treatment and control, and what is a valid alternative?

    • aA t-test assumes normally distributed individual observations, which latency violates; instead, use a Mann-Whitney U test on the raw latency values.
    • bThe sampling distribution of p99 is approximately normal, so a z-test for quantiles is valid without estimating the local density at the 99th percentile.
    • cThe Central Limit Theorem does not apply to order statistics, but you can preserve validity by applying the t-test only to the top one percent of observations.
    • dA t-test targets the sample mean, whereas p99 is an order statistic whose variance depends on local tail density; use bootstrap confidence intervals or permutation tests.
    Show the answer

    Answer: d · A t-test targets the sample mean, whereas p99 is an order statistic whose variance depends on local tail density; use bootstrap confidence intervals or permutation tests.

    A t-test is derived for the sample mean under the CLT, while p99 is an order statistic whose standard error is driven by the local density at the tail, making resampling methods like bootstrap confidence intervals or permutation tests the valid approach. Option C is tempting because it correctly notes the CLT does not apply to order statistics, but isolating the top one percent destroys the overall sample structure and does not fix the fundamental mismatch between a mean test and a quantile estimate.

    Read the full bite: Why can't you t-test p99 latency, and what's a valid alternative?

28. Question 28 of 30

    Why must user variant assignment remain fixed throughout an email A/B test rather than re-randomizing daily?

    • aIt eliminates the need for salt-based hashing in the bucketing service.
    • bIt ensures the email provider delivers messages with exactly-once semantics.
    • cIt prevents users from seeing both variants and polluting statistical independence.
    • dIt allows mid-test traffic shifting to the better-performing subject line.
    Show the answer

    Answer: c · It prevents users from seeing both variants and polluting statistical independence.

    Deterministic bucketing locks each user to a single variant, keeping observations independent for valid hypothesis testing, whereas re-randomization lets users flip groups and contaminates results. Mid-test traffic shifting introduces peeking bias, and exactly-once semantics is an event-tracking concern unrelated to bucketing.

    Read the full bite: Architect email subject line A/B testing for a large user base

29. Question 29 of 30

    Which combination of randomization, instrumentation, and planning best ensures a rigorous A/B test for a redesigned dashboard?

    • aAssign variants per device ID to capture all traffic, log exposure and outcomes on the backend, and calculate the required sample size using baseline traffic.
    • bUse cookie-based assignment for consistent UX, track a primary engagement metric and page load time, and run the test for two weeks.
    • cRandomize users with a stable hash of user ID, measure the primary metric via frontend pings, and monitor the experiment daily without a fixed stopping date.
    • dHash user IDs for stable assignment, log exposure on the frontend and business outcomes on the backend, define an MDE to pre-calculate runtime, and monitor guardrail metrics.
    Show the answer

    Answer: d · Hash user IDs for stable assignment, log exposure on the frontend and business outcomes on the backend, define an MDE to pre-calculate runtime, and monitor guardrail metrics.

    Stable user-level hashing prevents treatment dilution, backend logging ensures reliable measurement, and pre-calculating runtime with guardrails protects against bias. Option C is tempting because it uses the correct randomization method, but relying solely on frontend pings risks data loss and daily peeking without a fixed stopping rule inflates false-positive rates.

    Read the full bite: How would you implement an A/B test for a redesigned dashboard?

30. Question 30 of 30

    An A/B test returns a p-value of 0.03. Which statement accurately describes its meaning under the null hypothesis?

    • aThere is a 97% probability that the variant is genuinely better than the control.
    • bThere is a 3% probability that the null hypothesis of no effect is actually true.
    • cThere is a 97% probability that this result will replicate in a future experiment.
    • dIf the null hypothesis of no difference were true, we would observe a result at least this extreme about 3% of the time.
    Show the answer

    Answer: d · If the null hypothesis of no difference were true, we would observe a result at least this extreme about 3% of the time.

    A p-value of 0.03 means that if there were truly no difference between variants, random variation would produce a result this extreme roughly 3% of the time. It does not mean there is a 97% chance the variant is better, because the p-value describes the probability of the data under the null rather than the probability that the null is false.

    Read the full bite: Explain what a p-value represents in an A/B test and interpret 0.03