Design a pricing-page A/B test for $10 vs $15 plans
This tests sticky bucketing and revenue attribution. Hash user IDs to lock variants, persist assignments server-side, and join experiment logs to subscriptions via shared IDs. Red flag: client-side randomization that flickers or breaks billing context.
What's really being asked
The interviewer wants to know if you can build a split experiment that is both consistent for the user and trustworthy for the business. This means understanding randomization, persistence, and attribution. Pricing experiments are high stakes because showing different prices to the same user destroys trust, and misattributing revenue breaks the experiment.
The full answer
A good answer hits four things in order. First, deterministic bucketing. Hash a stable identifier like a user ID or a device ID with the experiment ID to produce a bucket. This guarantees the same user always lands in the same variant without needing to store every assignment. Second, persistence. Cache the bucket server-side in the user profile or session store, and mirror it in a first-party cookie or local storage for anonymous users. This prevents price flicker on reload and survives logout. Third, logging. Emit an experiment exposure event to an immutable log the moment the user is bucketed. Include the experiment ID, variant ID, user ID, timestamp, and any context like device type. Fourth, attribution. Join the exposure event to the subscription event in your analytics warehouse using the same user ID or anonymous ID. Define an attribution window such as seven or thirty days so late conversions still count but are bounded.
The mistakes people make
A red flag is client-side only randomization with JavaScript that runs after the page renders. This causes price flicker and can be blocked by ad blockers or privacy settings. Another red flag is storing the assignment in a plain cookie without server-side validation, which means a user can clear cookies, see a new price, and pay under a different bucket. A third red flag is forgetting to log the exposure event and instead inferring the bucket from the page the user saw. This breaks if the user shares a link or if the page is cached by a CDN.
What usually comes next
The interviewer may ask how you handle users who visit while logged out and subscribe after logging in. You should mention anonymous ID stitching. They may ask about network effects or cannibalization between plans, which points to a longer holdout group or segment analysis. They may also ask about statistical significance and how long to run the test, which ties to baseline conversion rate, minimum detectable effect, and traffic volume.
A concrete example
Imagine a user visits the pricing page on Monday. Your edge function hashes their anonymous ID and the experiment ID pricing2024q3 modulo 100. They land in bucket B, the fifteen dollar plan. The edge function sets a first-party cookie with a signed payload containing the variant and experiment ID, and your backend logs an exposure event to Kafka. The user leaves and returns on Wednesday while logged in. Your auth system links the anonymous ID to their user ID, reads the signed cookie, and still shows the fifteen dollar plan. On Friday they subscribe. The subscription service emits a revenue event with the user ID. Your nightly ETL joins the earliest exposure event for that user to the subscription event within a fourteen day window and attributes the revenue to variant B.
Interview question
Which strategy best keeps pricing consistent across visits while avoiding a database row for every user-variant pair?
- a.Store the variant in a plain browser cookie and read it on each visit
- b.Assign the variant randomly in client-side JavaScript after the pricing page renders
- c.Hash the stable user ID with the experiment ID, cache server-side, and mirror in a signed first-party cookieCorrect
- d.Infer the variant from the page URL and skip a dedicated exposure log
Why? this is the answer
Deterministic hashing recomputes the same variant without needing to store every assignment, while server-side caching and a signed cookie prevent price flicker and survive logout or cookie clearing. A plain cookie alone is a red flag because users can clear it, see a new price, and pay under a different bucket, destroying trust and the experiment.
Just read this? Test yourself on what you have been reading.
Read the original → optimizely.com
- #ab testing
- #experimentation
- #pricing
- #product strategy
- #attribution
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles