Components of a testable A/B hypothesis

WHAT THIS TESTS This checks experimental discipline: whether you can state a prediction precise enough to be wrong, which is the difference between a real experiment and after-the-fact storytelling.

A GOOD ANSWER COVERS Five components. The change, stated concretely. The population it applies to. A single primary metric chosen before launch, with a predicted direction. A rationale rooted in a behavioral mechanism, why you expect the effect. A decision criterion, the minimum detectable effect and significance threshold that determines ship or not. Crucially the hypothesis is falsifiable: a clear result would disprove it. Good practice also pre-registers guardrail metrics so a primary win that harms something else is caught.

COMMON WRONG ANSWERS 'Changing the button will improve engagement,' which names no specific metric, no direction, and no threshold. Picking the primary metric after seeing results. Listing ten metrics with no primary, enabling cherry-picking. No rationale, so a win cannot be generalized.

LIKELY FOLLOW-UPS How do you choose the minimum detectable effect and required sample size? Why pre-register the primary metric? How do guardrails interact with the primary?

ONE CONCRETE EXAMPLE Weak: changing the CTA text will get more clicks. Strong: for logged-out visitors on the pricing page, changing the CTA from Sign Up to Start Free Trial will increase trial-start conversion rate, the primary metric, by at least 3 percent relative, because emphasizing free and trial lowers perceived commitment; we will ship if the lift is at least 3 percent at 95 percent confidence and bounce rate, a guardrail, does not rise.