Components of a testable A/B hypothesis
experimental rigor before launch.
a specific change, a predicted directional effect on one primary metric, a rationale, and a measurable success threshold.
What's really being asked
This checks experimental discipline: whether you can state a prediction precise enough to be wrong, which is the difference between a real experiment and after-the-fact storytelling.
The full answer
Five components. The change, stated concretely. The population it applies to. A single primary metric chosen before launch, with a predicted direction. A rationale rooted in a behavioral mechanism, why you expect the effect. A decision criterion, the minimum detectable effect and significance threshold that determines ship or not. Crucially the hypothesis is falsifiable: a clear result would disprove it. Good practice also pre-registers guardrail metrics so a primary win that harms something else is caught.
The mistakes people make
'Changing the button will improve engagement,' which names no specific metric, no direction, and no threshold. Picking the primary metric after seeing results. Listing ten metrics with no primary, enabling cherry-picking. No rationale, so a win cannot be generalized.
What usually comes next
How do you choose the minimum detectable effect and required sample size? Why pre-register the primary metric? How do guardrails interact with the primary?
A concrete example
Weak: changing the CTA text will get more clicks. Strong: for logged-out visitors on the pricing page, changing the CTA from Sign Up to Start Free Trial will increase trial-start conversion rate, the primary metric, by at least 3 percent relative, because emphasizing free and trial lowers perceived commitment; we will ship if the lift is at least 3 percent at 95 percent confidence and bounce rate, a guardrail, does not rise.
Interview question
What single property most distinguishes a strong A/B hypothesis from a weak one?
- a.It chooses the winning metric after observing the data
- b.It lists as many candidate metrics as possible
- c.It is falsifiable, naming a specific primary metric, direction, and decision threshold in advanceCorrect
- d.It avoids stating a behavioral rationale to stay unbiased
Why? this is the answer
Falsifiability via a pre-declared primary metric, direction, and threshold makes the experiment conclusive. Listing many metrics or picking them post hoc enables cherry-picking, and a rationale aids generalization.
Just read this? Test yourself on what you have been reading.
- #ab-testing
- #hypothesis
- #experimentation
- #metrics
- #rigor
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on ab-testing — each one lists the topics its interview covers.
See open roles