Bayesian vs. Frequentist A/B Testing

WHY IT EXISTS: A/B testing exists because intuition fails when comparing two product variants. Random noise and sampling variation can make a worse variant look better in a small group. You need a principled rule for deciding when the data truly supports a change. Frequentist and Bayesian frameworks offer two different rules for the same raw data.

THE MENTAL MODEL: Think of a frequentist as a skeptic who assumes the new variant is identical to the control. They ask how often they would see a gap this large by chance if they ran the experiment a thousand times with no real difference. The answer is the p-value. A Bayesian starts with an initial belief about how much lift is likely, then updates that belief as data arrives. They directly output a statement such as there is an 85 percent chance that variant B is better.

HOW IT WORKS: In a frequentist test, you choose a sample size in advance based on the minimum detectable effect and desired power. After collecting exactly that many observations, you compute a p-value. If you peek early and stop the moment the p-value crosses a threshold, you inflate the false positive rate because multiple comparisons accumulate chance fluctuations. In a Bayesian test, you define a prior distribution representing your belief about the conversion rate before seeing data. As observations arrive, Bayes theorem updates this to a posterior distribution. You can check the posterior at any time and stop when the probability of B beating A exceeds a threshold such as 95 percent. The validity of this flexibility depends entirely on the prior being reasonable.

WHEN TO USE IT: Use frequentist methods when your organization demands strict error control, when regulators or leadership require pre-registered sample sizes, or when you want to avoid arguments about subjective assumptions. Use Bayesian methods when you have genuine prior data from past experiments, when you need to communicate results to stakeholders who think in probabilities rather than p-values, or when the cost of waiting for a fixed sample size outweighs the risk of a slightly higher error rate.

WHEN NOT TO USE IT: Do not use frequentist peeking without a sequential testing correction such as a spending function, because naive early stopping destroys the meaning of the p-value. Do not use Bayesian testing with a default or uninformed prior just to stop early; a prior that is too narrow or centered on an optimistic lift can pull the conclusion away from what the data actually shows, especially in the first few hundred observations.

ONE CANONICAL EXAMPLE: Imagine you are testing a new checkout button color. After two thousand users, the new color shows a 5 percent relative lift in conversions. A frequentist analyst would say that if the button truly made no difference, the chance of observing a lift this large or larger is 3 percent. They compare that to a pre-defined alpha of 5 percent and reject the null hypothesis. A Bayesian analyst with a prior centered on zero lift and moderate variance would say that given the data and the prior, there is a 94 percent probability that the new color converts better. They might ship it if the posterior probability crosses their decision threshold.