Design a contamination-safe pricing experiment

WHAT THIS TESTS This checks whether you grasp that pricing experiments carry both statistical interference and a fairness or trust hazard that ordinary A/B tests do not. The interviewer wants a randomization unit that prevents users from seeing different prices side by side, plus honest treatment of the analysis trade-offs.

A GOOD ANSWER COVERS Start by naming the dual risk: if two users on different prices compare notes, the lower-priced user is fine but the higher-priced one feels cheated, driving churn and PR risk, and the comparison itself contaminates behavior. The fix is to randomize at a unit where everyone inside sees one price. Geo-based testing assigns whole markets, such as countries or metro areas, to the new or old price, so neighbors and social circles largely share a price. Time-based cohorts apply the new price to everyone who signs up after a cutoff, grandfathering existing users, which is common and defensible because users expect prices to change for new customers. Either way you analyze at the cluster or cohort level: compare revenue per user and churn between geos or between cohorts, not between individuals, since within-unit users are correlated. Control for confounds, geos differ in baseline behavior and cohorts differ in seasonality, using pre-period baselines or difference-in-differences. Acknowledge lower power from fewer units.

COMMON WRONG ANSWERS Running a straight user-level price test and ignoring fairness backlash. Showing different prices to users who can clearly compare, like members of the same team. Analyzing a geo experiment at the user level, understating variance. Forgetting that geos and cohorts have baseline differences that must be controlled.

LIKELY FOLLOW-UPS How do you control for geos being inherently different? Difference-in-differences against a pre-period. Why grandfather existing users? Fairness and retention. How many geos do you need for power? Depends on between-geo variance.

ONE CONCRETE EXAMPLE A SaaS company tests a twenty percent price rise using time-based cohorts: everyone signing up after a date sees the new price, existing customers keep theirs. After a quarter the team compares the new cohort's revenue per user and churn against the prior cohort, adjusting for seasonality with a difference-in-differences baseline. No existing customer sees a surprise increase, eliminating the side-by-side comparison risk while still measuring impact.