Design an A/B test for a 'Buy Now' button color change

WHAT THIS TESTS: Your ability to translate a vague product idea into a disciplined experimental plan. Interviewers want to see that you understand the full lifecycle: hypothesis formation, metric hierarchy, statistical design, execution, and interpretation. At the senior level they also watch for whether you protect against false positives and business harm.

A GOOD ANSWER COVERS: Start with a falsifiable hypothesis. Example: changing the Buy Now button from grey to red will increase purchase conversion by at least two percent relative to the control. Next, pick a primary metric that maps to business value. Purchase conversion rate is better than click-through rate because a click without a purchase is a hollow signal. Then choose guardrail metrics such as return rate, refund rate, or average order value to ensure the change does not cannibalize overall revenue. After that, size the experiment. State your baseline conversion, your minimum detectable effect, significance threshold, and power, then translate that into a sample size and calendar duration. Randomize at the user level, not session level, to avoid contamination. Pre-register your stopping rule: you will run for the full duration unless a guardrail metric crosses a safety boundary. Finally, outline the analysis plan: intent-to-treat, sanity checks on assignment ratios, and a decision framework for ship, iterate, or discard.

COMMON WRONG ANSWERS: Choosing click-through rate or button engagement as the primary metric without connecting it to purchases. Stopping the test as soon as statistical significance is reached rather than sticking to the planned duration. Ignoring segment effects such as mobile versus desktop. Failing to mention randomization unit or assuming you can just split traffic fifty-fifty without accounting for logged-out users or cookie churn.

LIKELY FOLLOW-UPS: How would you handle a low-traffic page where sample size would take months? What would you do if the primary metric is flat but a secondary metric moves significantly? How do you guard against multiple comparison problems when looking at many segments? Would you run this as an A/B/n test with several colors?

ONE CONCRETE EXAMPLE: Suppose baseline purchase conversion is five percent and you want to detect a ten percent relative lift. At ninety-five percent confidence and eighty percent power you need roughly sixty-two thousand users per variant. If the product page gets ten thousand unique users per day, you need about thirteen days. You would randomize by user_id, keep the allocation at fifty-fifty, and monitor refund rate as a guardrail. If refund rate rises more than one percentage point in the treatment, you pause the test immediately.