Design an A/B test for a Buy Now button

WHAT THIS TESTS: This question evaluates whether you can design a reliable experimentation platform rather than simply changing a CSS color. Interviewers want to see that you understand randomization mechanics, state persistence across sessions, and the telemetry pipeline required for trustworthy analysis. The scope spans frontend rendering, backend identity, and data engineering.

A GOOD ANSWER COVERS four components in order. First, deterministic assignment: hash the user ID concatenated with the experiment ID into an integer, then take modulo 100 to map into percentiles, so variant A might be 0 through 49 and variant B 50 through 99. This guarantees the same user always lands in the same bucket without storing a separate assignment table. Second, sticky persistence: if the user is logged in, store the variant in their profile table; if anonymous, write a first-party cookie with the experiment ID and variant, plus an expiration date aligned with the experiment duration. Third, rendering consistency: on page load, the application should read the persisted variant before rendering to avoid color flicker or layout shift; server-side rendering is preferred, but if client-side then hide the button until the variant resolves. Fourth, logging: emit an impression event when the button is painted with the variant, and a conversion event when the user clicks; both payloads must include the experiment ID, variant name, user ID, timestamp, and any pre-exposure covariates like device type or region.

COMMON WRONG ANSWERS include using Math.random in the browser without persistence, which reshuffles users on every visit and breaks statistical independence. Another red flag is logging only clicks without impressions, which makes it impossible to compute click-through rates or guard against sample ratio mismatch. Some candidates also forget to handle logged-out users or propose storing variants in localStorage without a fallback for cross-device consistency.

LIKELY FOLLOW-UPS include how to handle network failures in event logging, how to support mutually exclusive experiments so one user is not in ten tests at once, and how to calculate sample size or guardrail metrics like page load time. Interviewers may also ask how you would roll out the winning variant or clean up experiment code.

ONE CONCRETE EXAMPLE: Suppose you have one million daily active users and a 50-50 split. You hash each user ID with the string buy-now-color-exp-42 using MurmurHash3, modulo 100. Users with values 0-49 see blue; 50-99 see green. The variant is written to a cookie named exp_42 that expires in 14 days. When the product page renders, the server reads the cookie and injects the CSS class before sending HTML. An analytics beacon fires an impression event on paint and a conversion event on click, both batched and sent to a warehouse where analysts group by variant to compute lift with a two-proportion z-test.

Source: Wikipedia: A/B testing