Implementing a consistent-assignment A/B test

WHAT THIS TESTS Whether you can design an A/B system that assigns deterministically, logs exposure correctly, and supports a clean win or lose analysis.

A GOOD ANSWER COVERS Three core components. First, an assignment or feature-flag service that decides each user's variant. Second, exposure logging that records when a user was actually bucketed into the experiment and which variant they saw, which are the events defining the denominator. Third, event tracking for the conversion event purchase_complete, tied to the same identity. Consistency across sessions and devices is the crux: assignment must be deterministic, not random per request. Hash a stable identifier, the logged-in user id, together with the experiment key, take a modulo over the bucket space, and map ranges to variants. The same user id always hashes to the same bucket, so they see the same variant everywhere they log in, with no need to store every assignment, though caching assignments is also valid. When the user is anonymous, bucket on a persistent device or cookie id and re-stitch on login.

COMMON WRONG ANSWERS Assigning randomly per session, so a user flips variants between visits and contaminates both arms. Bucketing on session id or device only, breaking cross-device consistency. Measuring conversion over all users rather than only exposed users. Forgetting exposure logging, which makes the denominator wrong.

ANALYSIS QUERY Start from the exposure table, one row per user-experiment with the assigned variant, left join to purchase_complete events occurring after the exposure timestamp, then group by variant to compute count of converters over count of exposed users, yielding conversion rate per variant. Add a two-proportion z-test or chi-square to judge significance, and report confidence intervals, not just point estimates.

ONE CONCRETE EXAMPLE hash(user_id + checkout_v2) mod 100 sends 0 to 49 to control and 50 to 99 to treatment. The analysis groups exposed users by variant, divides purchasers by exposed, finds treatment at 8.1 percent versus control 7.2 percent, and a z-test confirms the lift is significant before full rollout.