Top 30 A/b testing Interview Questions and Answers
30 multiple-choice questions on A/b testing, drawn from 30 bites out of the 104 tagged A/b testing on Tezvyn. Answer them here or read straight down. Every question carries the correct option, why it is correct, and a link to the bite it came from.
30 questions. Pick an answer, or open “Show the answer” to read it.
Answers are graded in your browser. Nothing is saved, and no XP or streak is earned here. The app keeps score.
Question 1 of 30
A team redesigns a checkout flow, successfully increasing the conversion rate. Which guardrail metric is most critical for ensuring this change didn't inadvertently harm overall revenue?
Show the answer
Answer: a · Average Order Value (AOV)
While conversion rate measures purchase frequency, Average Order Value (AOV) measures the value of each purchase. A change could increase conversions of low-value carts, hurting overall revenue, which AOV would reveal. The other options are valid guardrails but do not directly measure the financial impact.
Read the full bite: Pitfalls of 'Conversion Rate' as a North Star Metric
Question 2 of 30
For which situation is a hypothesis-driven analysis LEAST appropriate?
Show the answer
Answer: c · Uncovering novel insights and potential trends within a newly acquired dataset.
Hypothesis-driven analysis is designed for testing specific, falsifiable statements. Uncovering novel insights from a new dataset is an open-ended discovery task, which is better suited for exploratory data analysis rather than hypothesis testing.
Read the full bite: Hypothesis-Driven Analysis: Ask First, Analyze Second
Question 3 of 30
In a hypothesis test with an alpha of 0.05, what is the correct interpretation if your p-value is 0.03?
Show the answer
Answer: c · You have sufficient evidence to reject the null hypothesis.
The card states that if the p-value is less than alpha (p < alpha), you 'reject the null hypothesis.' Since 0.03 is less than 0.05, the null hypothesis is rejected. The p-value is not the probability of the null hypothesis being true, which is a common misconception.
Read the full bite: Hypothesis Testing: A Courtroom for Your Data
Question 4 of 30
In A/B testing, when is the Minimum Detectable Effect (MDE) primarily determined and used?
Show the answer
Answer: d · Before the experiment begins, to calculate the required sample size for detecting a meaningful change.
The card explicitly states that MDE is an input for experiment design, determined 'before a test' to calculate the required sample size. It is a planning tool, not an analysis tool used after the experiment or during runtime.
Read the full bite: Minimum Detectable Effect: How Small a Change Can You See?
Question 5 of 30
What is the fundamental role of the null hypothesis in scientific testing?
Show the answer
Answer: a · To provide a default position of no effect or no difference that requires strong evidence to overturn.
The null hypothesis establishes a baseline assumption that there is no effect or difference, acting as a default that must be disproven with sufficient evidence. Option D describes the alternative hypothesis, which represents the effect the researcher aims to demonstrate.
Read the full bite: Null Hypothesis: Assume No Effect Until Proven Otherwise
Question 6 of 30
Without proper randomization, what is the main challenge in interpreting the results of an experiment?
Show the answer
Answer: b · Determining if the observed outcomes were truly caused by the intervention being tested.
Randomization's primary purpose is to ensure that experimental groups are comparable, allowing researchers to confidently attribute observed outcome differences to the intervention. Without it, pre-existing differences between groups (confounding variables) could be mistaken for the effect of the intervention, making causal claims unreliable. Option A, while important for statistical power, addresses the quantity of data, not the quality of causal inference.
Read the full bite: Randomization: The Key to Trustworthy Experiments
Question 7 of 30
Which scenario is most appropriate for applying a Student's t-test?
Show the answer
Answer: d · Determining if a new ad campaign significantly increased sales compared to the old campaign.
The t-test is specifically used to compare the means of two groups to determine if an observed difference is statistically significant, as described in option D. Option A is incorrect because the t-test is not suitable for comparing more than two groups.
Read the full bite: Student's t-test: Is This Difference Real or Just Noise?
Question 8 of 30
After a new UI feature launch, existing users show an initial dip in engagement, then recovery and improvement. What explains this?
Show the answer
Answer: c · A learning effect, as users adapted to the new interface.
The card defines the learning effect as an initial dip in metrics as users adapt to a change, followed by recovery and potential improvement, which perfectly matches the described pattern. The novelty effect, in contrast, would manifest as an initial temporary lift in metrics, not a dip.
Read the full bite: Novelty and Learning Effects in A/B Testing
Question 9 of 30
A sales team member achieves a record-breaking quarter. Based on the principle of regression to the mean, what is most likely to happen in the following quarter?
Show the answer
Answer: a · Their sales performance will likely decrease, moving closer to their historical average.
Regression to the mean posits that extreme outcomes, which are partly due to random chance, are unlikely to be repeated. Therefore, a record-breaking performance is most likely to be followed by a performance closer to the individual's average. Options B and C incorrectly assume the extreme performance establishes a new baseline or trend, while D overstates the expected regression.
Read the full bite: Regression to the Mean: Why Outliers Settle Down
Question 10 of 30
What is a primary drawback of applying the Bonferroni correction, especially with many comparisons?
Show the answer
Answer: a · It becomes excessively conservative, making it harder to detect real effects.
The correct answer is C because the card states that with many comparisons, the Bonferroni correction 'becomes so conservative that it dramatically increases your chance of a false negative (missing a real effect).' Option C is incorrect because the Bonferroni correction's purpose is to reduce the chance of a Type I error (incorrectly rejecting a true null hypothesis), not increase it.
Read the full bite: Bonferroni Correction: Raising the Bar for Significance
Question 11 of 30
Your team sees high checkout drop-off but does not know the cause. Which is the best first step?
Show the answer
Answer: b · Run usability tests to observe where users get stuck, then form hypotheses for a follow-up A/B test
Usability testing is the right first step because it uncovers why users struggle and generates hypotheses for later A/B validation. A/B testing compares live variants but cannot diagnose root causes or reveal why users drop off, so running it first on an unknown broken flow wastes traffic.
Read the full bite: Usability vs A/B testing: when should engineers advocate for each?
Question 12 of 30
A strong correlation is found between Feature X usage and user retention. What is the most rigorous initial step to assess if investing more in Feature X will truly boost retention?
Show the answer
Answer: a · First, scrutinize the data for statistical significance, clarify metric definitions, and investigate potential confounding variables like inherent user engagement.
Option A outlines the essential initial steps: questioning data validity, defining metrics, and identifying confounding variables. This crucial exploratory analysis should precede expensive experiments like A/B tests (Option D) or more specific analyses like cohort analysis (Option B), and it moves beyond simply stating the 'correlation isn't causation' cliché (Option C).
Read the full bite: Feature A correlates with retention. Should we invest more?
Question 13 of 30
A stakeholder observes users of a feature have higher retention and suggests promoting it to everyone. What is the best initial step to investigate their claim?
Show the answer
Answer: a · Acknowledge the finding and analyze user cohorts to see if these users were already more engaged before adopting the feature.
The correct approach is to first investigate confounding variables through cheaper analyses like cohort analysis. This is more pragmatic than immediately launching an A/B test, which is resource-intensive.
Read the full bite: Stakeholder claims correlation implies causation. How do you investigate?
Question 14 of 30
Why might a gradient plot be a better choice than a standard error bar for visualizing the uncertainty of an A/B test result?
Show the answer
Answer: c · It visually conveys that outcomes near the mean are more probable than outcomes at the ends of the confidence interval.
A gradient plot's fading color intensity accurately shows that outcomes are more probable near the mean, correcting a common misinterpretation of standard error bars. Distractor A is tempting but describes a violin plot, not a gradient plot.
Read the full bite: How would you visually represent statistical uncertainty in a chart?
Question 15 of 30
When presenting A/B test results, what is the key benefit of supplementing static confidence intervals with hypothetical outcome plots?
Show the answer
Answer: d · HOPs make uncertainty tangible by showing that the observed leader can lose in plausible alternative samples
The card states that hypothetical outcome plots make abstract uncertainty concrete by showing possible values rather than a single static interval, such as when the 14 percent bar sometimes drops below 12 percent. Option B is tempting because visual clutter is a real concern with many intervals, but the card notes that animation can introduce its own cognitive biases rather than simply replacing clutter.
Read the full bite: How would you visually represent statistical uncertainty in a chart?
Question 16 of 30
To best communicate the full probability density of a metric's potential outcomes when comparing two groups in an A/B test, which visualization is most appropriate?
Show the answer
Answer: d · A violin plot for each group
Violin plots are specifically highlighted in the card as showing the 'full probability density' and providing a 'much richer picture of the uncertainty' compared to other methods. While confidence intervals and box plots show aspects of uncertainty or distribution, they do not convey the entire probability density shape as effectively as a violin plot.
Read the full bite: How do you visually represent statistical uncertainty in a chart?
Question 17 of 30
A product manager notes that users who engage with a new "Community Forum" feature show 15% higher 30-day retention. What is the most effective way to establish if the forum *causes* this increased retention?
Show the answer
Answer: c · Implement a randomized A/B test, assigning new users to either a control group without the forum or a treatment group with prominent forum access, then compare their retention.
The card highlights that correlation is not causation and identifies a randomized A/B test as the gold standard for proving causality. Option C describes such an experiment, directly testing the feature's causal impact. Option B incorrectly assumes causality from correlation, a common fallacy the card warns against.
Read the full bite: Is 20% higher retention from Feature X causal or correlational?
Question 18 of 30
Users who adopt a new 'Advanced Search' feature show 20% higher retention. What is the most reliable first step to determine the feature's actual impact on retention?
Show the answer
Answer: d · Run an A/B test on new users, showing the feature to one randomly assigned group.
The observed 20% lift is a correlation, likely due to selection bias (more engaged users adopt the feature). An A/B test is the best way to establish causality by isolating the feature's effect from pre-existing user characteristics.
Read the full bite: Is Feature X Causal for 20% Higher Retention?
Question 19 of 30
An A/B test shows a 5% lift in conversion, but traffic is imbalanced (80% treatment, 20% control) and device distribution is skewed. What is the most appropriate immediate next step?
Show the answer
Answer: b · Declare the aggregate 5% lift untrustworthy, then segment the results by device to analyze performance within each segment.
The correct approach is to first invalidate the aggregate result due to confounding variables, then segment the data to salvage insights. Immediately re-running the test or only fixing the bug misses the opportunity to learn from the current, albeit flawed, experiment.
Read the full bite: An A/B test has imbalanced traffic. What do you do?
Question 20 of 30
If an A/B test for a new website feature yields a p-value of 0.02, what is the most accurate interpretation?
Show the answer
Answer: b · There is a 2% chance of observing results as extreme as these, assuming the new feature has no actual effect.
A p-value indicates the probability of seeing your results by chance, assuming the null hypothesis (no effect) is true. It does not state the probability of the null hypothesis being true, nor does it guarantee the alternative hypothesis.
Read the full bite: P-Value: Gauging Surprise, Not Certainty
Question 21 of 30
When designing an A/B test to measure how API latency impacts D7 retention, which setup best isolates causality?
Show the answer
Answer: d · Bucket users by user ID, inject a consistent server-side delay per bucket, and track D7 retention alongside P99 latency
User-level randomization with consistent server-side delays ensures the only systematic difference between groups is the injected latency, establishing causality. Comparing historical slow versus fast requests is confounded by variables like device type or network quality, so retention differences cannot be attributed solely to latency.
Read the full bite: How would you design an experiment measuring API latency impact on retention?
Question 22 of 30
An A/B test shows a 5% metric lift, but the treatment group has a much higher percentage of mobile users. What is the most appropriate immediate action?
Show the answer
Answer: c · Segment the results by device type to analyze the lift for mobile and desktop users separately.
The correct action is to segment the results to understand the feature's true impact on each device group, as the overall lift is likely misleading. Discarding the results without this analysis (Option A) misses a key learning opportunity, even though a re-run is the ultimate solution.
Read the full bite: A/B Test Results with Skewed Traffic: What's Next?
Question 23 of 30
When evaluating a feature's impact post-launch without an A/B test, and suspecting self-selection bias, which approach best isolates the feature's true effect?
Show the answer
Answer: d · Use Propensity Score Matching to construct a comparable control group from non-adopters based on pre-feature characteristics.
Propensity Score Matching (PSM) directly addresses self-selection bias by creating a synthetic control group of non-adopters with similar pre-feature characteristics, allowing for a more accurate causal comparison. Directly comparing adopters and non-adopters (Option A) is flawed due to this bias, as adopters are likely more engaged initially.
Read the full bite: How would you validate a PM's claim about feature success?
Question 24 of 30
A feature launched without an A/B test shows adopters have a much higher success metric. What is the most rigorous approach to estimate the feature's true causal impact?
Show the answer
Answer: a · Create a synthetic control group by matching adopters with non-adopters who had similar pre-feature characteristics, then compare their success metrics.
This approach, often using Propensity Score Matching, creates a comparable control group to isolate the feature's effect from pre-existing user differences. Simply segmenting by general activity is less rigorous as it doesn't control for the multiple factors that cause self-selection.
Read the full bite: How would you validate a feature's success, correcting for self-selection?
Question 25 of 30
In an A/B test with a p-value of 0.03 and an alpha of 0.05, which statement is the most accurate interpretation?
Show the answer
Answer: d · If the new feature truly had no effect, a result at least as extreme as this would occur 3% of the time.
A p-value of 0.03 means there is a 3% probability of observing a result at least as extreme as the one obtained, assuming the null hypothesis (that there's no real difference) is true. Option B incorrectly states the p-value as the probability of the difference being due to chance, rather than the probability of observing such a result under the null hypothesis.
Read the full bite: What does a p-value of 0.03 mean in an A/B test?
Question 26 of 30
An A/B test yields a p-value of 0.03. What is the most accurate interpretation of this result?
Show the answer
Answer: b · Assuming the feature has no real effect, there is a 3% chance of observing a result at least this extreme.
The p-value is a conditional probability. It's the chance of seeing a result this extreme *if* the null hypothesis were true. A common error is to misinterpret it as the direct probability that the null hypothesis is true or that the result was random.
Read the full bite: What does a p-value of 0.03 mean in an A/B test?
Question 27 of 30
Given a p-value of 0.08 and a significance level of 0.05 for a new feature, what is the best recommendation to a product manager?
Show the answer
Answer: d · Explain the 8% risk of a false positive, quantify potential business costs, and suggest extending the test for more data.
The best approach involves translating the p-value into a business risk (8% chance of a false positive), discussing the potential costs, and proposing a solution like extending the test to gather more data. Option B misinterprets the p-value as the probability of the hypothesis being true, which is a common misconception.
Read the full bite: p-value is 0.08, significance is 0.05. Ship it?
Question 28 of 30
An A/B test shows a p-value of 0.08. How should you advise a product manager who wants to ship the feature?
Show the answer
Answer: b · Frame the 8% false positive risk in terms of business cost, and suggest running the test longer to gather more conclusive data.
This approach correctly translates the p-value into business risk (8% chance of a false positive) and proposes a constructive path forward, balancing statistical rigor with business goals. Stating there's a 92% chance the feature is better is a common but incorrect interpretation of a p-value.
Read the full bite: A/B test p-value is 0.08, PM wants to ship. What now?
Question 29 of 30
An A/B test shows a new feature decreases overall user engagement, but engagement increases within every individual user segment. What is the most likely cause of this discrepancy?
Show the answer
Answer: c · The experiment suffered from a confounding variable unevenly distributed across test and control groups.
This scenario describes Simpson's Paradox, which occurs when a trend reverses between aggregate and segmented data due to an uneven distribution of a confounding variable. Option B is a common misconception, as the paradox indicates a flawed experiment design that requires re-evaluation, not simply ignoring the aggregate.
Read the full bite: Explain Simpson's Paradox with a user engagement example
Question 30 of 30
An A/B test shows a new feature has a lower overall success rate, but a higher success rate within every individual user segment. What is the most likely conclusion?
Show the answer
Answer: c · The feature is successful; the negative overall result is likely caused by an uneven mix of user segments in the test groups.
This describes Simpson's Paradox. The feature is a success for all users, but a confounding variable (the uneven mix of segments) makes the aggregate result misleading. The most tempting distractor is to trust the aggregate data, which is the primary mistake the paradox highlights.
Read the full bite: Explain Simpson's Paradox with a user engagement example
Could you explain these out loud?
That is what an interview actually tests. Tezvyn gives you questions like these with what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.