How would you A/B test a 'Buy Now' button color change?

WHAT THIS TESTS: This is a foundational question testing your understanding of the scientific method as applied to product development. Interviewers want to see if you can move beyond just 'let's test it' to a structured process. They're evaluating your ability to define a clear goal, select appropriate metrics to measure success, and consider potential negative consequences. It's a check for rigor and data-driven thinking, even for a simple change.

A GOOD ANSWER COVERS: A strong answer walks through four key design steps. First, formulate a clear, falsifiable hypothesis, such as: 'Changing the 'Buy Now' button from blue to high-contrast green will increase its click-through rate because green is more visually prominent and associated with go.' Second, select a single primary metric that directly measures the hypothesis; for this, button click-through rate (CTR) is the correct choice. Third, define guardrail metrics to monitor for unintended negative effects, like overall page conversion rate, page load time, or bounce rate. Fourth, briefly mention experiment setup details like calculating the required sample size and duration to achieve statistical power (e.g., 95% confidence, 80% power) and ensuring random assignment of users.

COMMON WRONG ANSWERS: The most common mistake is jumping straight into implementation ('I'd test green vs. blue') without first stating a hypothesis. A test without a hypothesis is just data dredging. Another major red flag is selecting multiple primary metrics (e.g., 'we'll measure CTR, conversion, and revenue'). This dramatically increases the chance of a false positive and shows a misunderstanding of statistical principles. Finally, focusing only on the primary metric while ignoring guardrails is a sign of tunnel vision; a feature that increases clicks but hurts overall conversion is a failure.

LIKELY FOLLOW-UPS: Expect questions like: 'What if you don't see a statistically significant result?' (The correct action is to conclude there's no detectable effect and stick with the control, as change introduces risk). Or, 'What if CTR goes up, but overall conversion rate goes down?' (This tests product sense; you should not ship the change, as the ultimate business goal was harmed). You might also be asked how to calculate the sample size, which involves knowing the baseline rate, desired minimum detectable effect (MDE), and statistical power/significance.

ONE CONCRETE EXAMPLE: Imagine our baseline button CTR is 2%. We want to detect at least a 10% relative lift, so our target is a 2.2% CTR (an absolute MDE of 0.2%). Using a standard sample size calculator with 95% significance and 80% power, we'd need approximately 157,000 users per variation. If the page gets 100,000 visitors per day, we'd run the test for just over 3 days to collect enough data for both control and variant groups. We would only ship the green button if it shows a statistically significant lift in CTR without a statistically significant drop in our primary guardrail metric, overall purchase conversion rate.