How does cookie clearing affect A/B results and consistency?

WHAT THIS TESTS: This question evaluates whether you understand that an A/B test is only valid if assignment is stable and independent. When a user clears cookies or switches browsers, they can be re-bucketed into a different variant, which breaks the fundamental assumption that each user receives exactly one treatment. The interviewer wants to see that you recognize this as a threat to internal validity, not just a tracking inconvenience, and that you can design systems that persist assignment beyond fragile client-side state.

A GOOD ANSWER COVERS: First, describe the impact. Re-bucketing means the same physical user may see both variants, so you lose the ability to attribute revenue or conversion to a single experience. It also inflates the unique visitor denominator and can introduce Sample Ratio Mismatch if clearance correlates with behavior. Second, propose architecture. Propose server-side assignment based on a deterministic hash of a stable identifier such as a user ID for authenticated sessions or a first-party cookie with extended expiration. If the user is anonymous, mention device fingerprinting derived from signals like IP plus user-agent entropy, or using localStorage and IndexedDB as persistent stores that survive cookie deletion. Third, discuss edge assignment through a CDN or API gateway so the variant decision happens before the page renders. Fourth, acknowledge privacy constraints: fingerprinting must respect consent and browser anti-tracking measures, and server-side tag management can centralize bucketing logic.

COMMON WRONG ANSWERS: Saying the effect is negligible because only a few users clear cookies. At scale, even five percent re-randomization can bias results. Proposing to simply increase cookie TTL without a fallback mechanism. Suggesting third-party cookies as a solution, which major browsers now block. Recommending IP-based bucketing alone, which breaks under NAT and ignores IPv6 rotation. Claiming that analytics can just de-duplicate users after the fact without explaining how you would deterministically link the two sessions.

LIKELY FOLLOW-UPS: How would you detect that re-bucketing is happening in production? What is your fallback if a user rejects all storage including localStorage? How do you handle cross-device consistency when a user starts on mobile and finishes on desktop? Would you exclude users with cleared storage from analysis, and how does that affect external validity? How does server-side assignment change your latency budget?

ONE CONCRETE EXAMPLE: Imagine a checkout page test where Version B offers a ten percent discount. A price-sensitive user sees Version A on Monday, clears cookies to hunt for coupons, and sees Version B on Tuesday with the discount. They convert. Your system credits Version B, but the user was actually a control-qualified shopper who would have converted anyway, inflating B's lift. To prevent this, you assign variants at the edge using a hashed user ID when logged in, or a fingerprint-derived token stored in both a first-party cookie and localStorage with a one-week TTL. If both are missing, the user hits a deterministic assignment endpoint that returns the same variant based on a stable device signature, ensuring consistency without relying solely on browser cookies.

Source: guessthetest.com