How do you set up a button color A/B test?

WHAT THIS TESTS: This question tests whether you treat experimentation as a structured decision-making framework rather than a casual tweak. The interviewer wants to see that you understand randomization, metric selection, and pre-registration of parameters before any code ships. At a senior level they also care if you consider user-level consistency and statistical rigor, but even at the beginner level the focus is on knowing the essential components that separate a valid test from noise.

A GOOD ANSWER COVERS: A good answer hits six things in order. First, state the hypothesis clearly, for example that changing the button from blue to green will increase sign-ups because green implies go or action. Second, define the primary metric, which is sign-up conversion rate, and clarify whether you mean click-through rate on the button or end-to-end account creation. Third, specify the randomization unit, which should be the user or session level with a consistent bucket so the same visitor always sees the same variant. Fourth, set the traffic split, typically fifty-fifty between control and variation, and ensure the assignment is truly random rather than based on geography or time of day. Fifth, pre-calculate sample size or runtime using baseline conversion rate, minimum detectable effect, and chosen power and significance levels, for instance eighty percent power and a five percent alpha. Sixth, document the decision criteria in advance, such as we will ship the green button only if it wins with statistical significance and the absolute lift is at least two percent.

COMMON WRONG ANSWERS: Common wrong answers include saying you will run the test for a week because that feels right, or stopping early as soon as the green variant looks better. Another red flag is ignoring the control group entirely and simply comparing this weeks green button against last weeks blue button, which invites seasonality bias. Some candidates also forget to define the primary metric up front and instead plan to look at every dashboard metric afterward, which is p-hacking. Finally, assigning users by device or browser without a stable identifier breaks the user experience and dilutes the measured effect.

LIKELY FOLLOW-UPS: A likely follow-up is how you would handle a low-traffic page where reaching the required sample size takes months. Another is what you would do if the sign-up metric improves but revenue per user drops. Interviewers may also ask how you would prevent flicker or layout shift when swapping the button color, or whether you would run an A/A test first to validate your instrumentation.

ONE CONCRETE EXAMPLE: Imagine your current blue button has a baseline sign-up rate of ten percent and you want to detect a lift to eleven percent. With a five percent significance level and eighty percent power, you need roughly thirty thousand users per variant. You randomize by user ID, run for exactly fourteen days to cover two full weekly cycles, and pre-register that you will only ship green if the two-tailed test shows p less than zero point zero five and the directional lift holds in a holdout validation. This structure turns an opinion about color psychology into a data-informed decision.

Source: optimizely.com