Design a contamination-safe pricing experiment
choosing a randomization unit that hides price differences.
user-level price tests leak via fairness perception, so use geo holdouts or time-based cohorts where everyone in a unit sees one price.
What's really being asked
This checks whether you grasp that pricing experiments carry both statistical interference and a fairness or trust hazard that ordinary A/B tests do not. The interviewer wants a randomization unit that prevents users from seeing different prices side by side, plus honest treatment of the analysis trade-offs.
The full answer
Start by naming the dual risk: if two users on different prices compare notes, the lower-priced user is fine but the higher-priced one feels cheated, driving churn and PR risk, and the comparison itself contaminates behavior. The fix is to randomize at a unit where everyone inside sees one price. Geo-based testing assigns whole markets, such as countries or metro areas, to the new or old price, so neighbors and social circles largely share a price. Time-based cohorts apply the new price to everyone who signs up after a cutoff, grandfathering existing users, which is common and defensible because users expect prices to change for new customers. Either way you analyze at the cluster or cohort level: compare revenue per user and churn between geos or between cohorts, not between individuals, since within-unit users are correlated. Control for confounds, geos differ in baseline behavior and cohorts differ in seasonality, using pre-period baselines or difference-in-differences. Acknowledge lower power from fewer units.
The mistakes people make
Running a straight user-level price test and ignoring fairness backlash. Showing different prices to users who can clearly compare, like members of the same team. Analyzing a geo experiment at the user level, understating variance. Forgetting that geos and cohorts have baseline differences that must be controlled.
What usually comes next
How do you control for geos being inherently different? Difference-in-differences against a pre-period. Why grandfather existing users? Fairness and retention. How many geos do you need for power? Depends on between-geo variance.
A concrete example
A SaaS company tests a twenty percent price rise using time-based cohorts: everyone signing up after a date sees the new price, existing customers keep theirs. After a quarter the team compares the new cohort's revenue per user and churn against the prior cohort, adjusting for seasonality with a difference-in-differences baseline. No existing customer sees a surprise increase, eliminating the side-by-side comparison risk while still measuring impact.
Interview question
Why is a user-level A/B test especially problematic for testing a price increase, beyond normal statistical concerns?
- a.User-level tests always require more traffic than geo tests
- b.Prices cannot be changed dynamically for individual users
- c.Revenue cannot be measured at the user level
- d.Users on different prices may compare them, causing fairness backlash and contaminated behaviorCorrect
Why? this is the answer
Price is visible and emotionally charged, so users who discover unequal pricing churn or change behavior, biasing results and risking trust. Prices can be set per user technically and revenue is measurable per user; the unique issue is the fairness-driven contamination.
Just read this? Test yourself on what you have been reading.
Read the original → statsig.com
- #pricing-experiments
- #geo-testing
- #cohort-design
- #interference
- #difference-in-differences
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles