Design a pricing-page A/B test for $10 vs $15 plans

WHAT THIS TESTS: The interviewer wants to know if you can build a split experiment that is both consistent for the user and trustworthy for the business. This means understanding randomization, persistence, and attribution. Pricing experiments are high stakes because showing different prices to the same user destroys trust, and misattributing revenue breaks the experiment.

A GOOD ANSWER COVERS: A good answer hits four things in order. First, deterministic bucketing. Hash a stable identifier like a user ID or a device ID with the experiment ID to produce a bucket. This guarantees the same user always lands in the same variant without needing to store every assignment. Second, persistence. Cache the bucket server-side in the user profile or session store, and mirror it in a first-party cookie or local storage for anonymous users. This prevents price flicker on reload and survives logout. Third, logging. Emit an experiment exposure event to an immutable log the moment the user is bucketed. Include the experiment ID, variant ID, user ID, timestamp, and any context like device type. Fourth, attribution. Join the exposure event to the subscription event in your analytics warehouse using the same user ID or anonymous ID. Define an attribution window such as seven or thirty days so late conversions still count but are bounded.

COMMON WRONG ANSWERS: A red flag is client-side only randomization with JavaScript that runs after the page renders. This causes price flicker and can be blocked by ad blockers or privacy settings. Another red flag is storing the assignment in a plain cookie without server-side validation, which means a user can clear cookies, see a new price, and pay under a different bucket. A third red flag is forgetting to log the exposure event and instead inferring the bucket from the page the user saw. This breaks if the user shares a link or if the page is cached by a CDN.

LIKELY FOLLOW-UPS: The interviewer may ask how you handle users who visit while logged out and subscribe after logging in. You should mention anonymous ID stitching. They may ask about network effects or cannibalization between plans, which points to a longer holdout group or segment analysis. They may also ask about statistical significance and how long to run the test, which ties to baseline conversion rate, minimum detectable effect, and traffic volume.

ONE CONCRETE EXAMPLE: Imagine a user visits the pricing page on Monday. Your edge function hashes their anonymous ID and the experiment ID pricing2024q3 modulo 100. They land in bucket B, the fifteen dollar plan. The edge function sets a first-party cookie with a signed payload containing the variant and experiment ID, and your backend logs an exposure event to Kafka. The user leaves and returns on Wednesday while logged in. Your auth system links the anonymous ID to their user ID, reads the signed cookie, and still shows the fifteen dollar plan. On Friday they subscribe. The subscription service emits a revenue event with the user ID. Your nightly ETL joins the earliest exposure event for that user to the subscription event within a fourteen day window and attributes the revenue to variant B.

Source: Optimizely