How would you A/B test sign-up button copy and measure results?

WHAT THIS TESTS: This question tests whether you understand the full lifecycle of a controlled experiment in a production environment. It is not enough to say you would show half the users one button and half the other. The interviewer wants to see that you think about randomization, instrumentation, data integrity, and statistical validity. At a senior level, they also want to know you can avoid common pitfalls like selection bias, flicker, or measuring the wrong metric.

A GOOD ANSWER COVERS: A good answer hits four things in order. First, randomization and bucketing. You need a mechanism to randomly split traffic, such as hashing a user ID or session ID modulo 100, and you must ensure the assignment is sticky so the same user always sees the same variant. Second, rendering the variant. The copy swap should happen before the page paints to avoid flicker or layout shift, which means doing it server-side or immediately in the head of the document if client-side. Third, data collection. You must track at least two events: an impression when the button is shown, and a conversion when the user completes sign-up. Without the denominator of impressions, you cannot calculate a conversion rate. Fourth, analysis. You compare conversion rates between the two groups and use a statistical significance test, typically aiming for a p-value below 0.05 and ensuring the sample size is large enough to detect a meaningful effect.

COMMON WRONG ANSWERS: A common wrong answer is suggesting you show version A for one week and version B for the next week. This introduces temporal bias because traffic composition, day-of-week effects, or external campaigns change over time. Another red flag is only tracking clicks on the button rather than completed sign-ups, which creates a leaky funnel metric. Some candidates also forget to mention randomization entirely and suggest manual assignment or geo-based splitting, which breaks the control needed for causal inference.

LIKELY FOLLOW-UPS: An interviewer might ask how you would handle a user switching devices, which breaks sticky assignment unless you use a logged-in user ID. They might also ask how to run multiple concurrent experiments without interaction effects, or how you would calculate the required sample size before launching. Another follow-up is what you would do if the variant wins but the absolute lift is tiny, questioning business significance versus statistical significance.

ONE CONCRETE EXAMPLE: Suppose you have 100,000 daily visitors to the sign-up page. You hash each visitor's user ID to a number from 1 to 100. If the result is 50 or below, they see the control copy Start Free Trial; above 50, they see the variant Get Started Now. Your analytics pipeline fires an impression event when the button renders and a conversion event when the sign-up API returns success. After two weeks, the control has 50,000 impressions and 2,500 conversions for a 5.0 percent rate, while the variant has 50,000 impressions and 2,750 conversions for a 5.5 percent rate. You run a chi-squared test and find the result is statistically significant at p equals 0.03, so you recommend shipping the variant.

Source: optimizely.com