Compare A/B/n testing with multi-armed bandits for headline optimization

WHAT THIS TESTS: This question probes whether you can reason about the explore-exploit trade-off at a systems level. The interviewer wants to see that you understand A/B/n testing and multi-armed bandits are not interchangeable drop-in replacements but rather tools with different statistical guarantees, engineering footprints, and business outcomes. Senior candidates should demonstrate comfort with statistical power, regret minimization, and real-time infrastructure.

A GOOD ANSWER COVERS: Four things in order. First, allocation mechanics: A/B/n uses fixed traffic splits with a predetermined sample size and waits for a final analysis, while a multi-armed bandit dynamically shifts traffic toward better-performing variants using algorithms like epsilon-greedy, Upper Confidence Bound, or Thompson Sampling. Second, objective functions: A/B/n optimizes for unbiased inference and detecting small effect sizes with high confidence, whereas MABs optimize for cumulative reward during the experiment itself, accepting some bias in exchange for higher immediate conversion. Third, speed and sample efficiency: A/B/n requires waiting for the full sample to reach significance, which can be slow; MABs can identify winners faster in terms of opportunity cost but may never reach the same level of statistical certainty. Fourth, engineering complexity: A/B/n is stateless and easy to implement with simple randomization; MABs require stateful real-time decisioning, model updates, and operational monitoring, plus you must choose and tune an algorithm.

COMMON WRONG ANSWERS: Treating MABs as a strict upgrade to A/B/n in every scenario. Claiming that MABs give the same statistical power with less traffic. Ignoring the engineering cost of real-time allocation at scale. Failing to mention that early exploitation in MABs makes it hard to measure long-term effects or seasonal interactions because the losing arms are starved of traffic. Another red flag is conflating standard multi-armed bandits with contextual bandits; if you bring up personalization, you must show you know context requires visitor-specific data and a more complex model.

LIKELY FOLLOW-UPS: The interviewer might ask when you would still choose A/B/n over a bandit, how you would handle a non-stationary conversion rate that changes over time, or what metrics you would use to evaluate bandit performance beyond simple conversion rate. They may also ask about contextual bandits and whether you would add user segmentation or device-type context to the headline optimization problem.

ONE CONCRETE EXAMPLE: Imagine optimizing a checkout-page headline with three variants. In an A/B/n test, you assign 33 percent of traffic to each variant for two weeks, run a t-test at the end, and roll out the winner to 100 percent. Total opportunity cost is high because the losing variants get heavy traffic throughout. In an epsilon-greedy bandit with epsilon set to 0.1, you explore randomly only 10 percent of the time and exploit the current best headline 90 percent of the time. After one week the bandit may have shifted 80 percent of traffic to the winning headline, reducing regret but making it impossible to get a clean post-hoc read on the other two variants with any statistical confidence.

Source: optimizely.com