How does cookie clearing affect A/B results and consistency?
This tests bucketing integrity when state changes re-randomize users. Explain that re-bucketing contaminates the sample and triggers SRM; propose deterministic server-side assignment, fingerprinting, or authenticated IDs.
What's really being asked
This question evaluates whether you understand that an A/B test is only valid if assignment is stable and independent. When a user clears cookies or switches browsers, they can be re-bucketed into a different variant, which breaks the fundamental assumption that each user receives exactly one treatment. The interviewer wants to see that you recognize this as a threat to internal validity, not just a tracking inconvenience, and that you can design systems that persist assignment beyond fragile client-side state.
The full answer
First, describe the impact. Re-bucketing means the same physical user may see both variants, so you lose the ability to attribute revenue or conversion to a single experience. It also inflates the unique visitor denominator and can introduce Sample Ratio Mismatch if clearance correlates with behavior. Second, propose architecture. Propose server-side assignment based on a deterministic hash of a stable identifier such as a user ID for authenticated sessions or a first-party cookie with extended expiration. If the user is anonymous, mention device fingerprinting derived from signals like IP plus user-agent entropy, or using localStorage and IndexedDB as persistent stores that survive cookie deletion. Third, discuss edge assignment through a CDN or API gateway so the variant decision happens before the page renders. Fourth, acknowledge privacy constraints: fingerprinting must respect consent and browser anti-tracking measures, and server-side tag management can centralize bucketing logic.
The mistakes people make
Saying the effect is negligible because only a few users clear cookies. At scale, even five percent re-randomization can bias results. Proposing to simply increase cookie TTL without a fallback mechanism. Suggesting third-party cookies as a solution, which major browsers now block. Recommending IP-based bucketing alone, which breaks under NAT and ignores IPv6 rotation. Claiming that analytics can just de-duplicate users after the fact without explaining how you would deterministically link the two sessions.
What usually comes next
How would you detect that re-bucketing is happening in production? What is your fallback if a user rejects all storage including localStorage? How do you handle cross-device consistency when a user starts on mobile and finishes on desktop? Would you exclude users with cleared storage from analysis, and how does that affect external validity? How does server-side assignment change your latency budget?
A concrete example
Imagine a checkout page test where Version B offers a ten percent discount. A price-sensitive user sees Version A on Monday, clears cookies to hunt for coupons, and sees Version B on Tuesday with the discount. They convert. Your system credits Version B, but the user was actually a control-qualified shopper who would have converted anyway, inflating B's lift. To prevent this, you assign variants at the edge using a hashed user ID when logged in, or a fingerprint-derived token stored in both a first-party cookie and localStorage with a one-week TTL. If both are missing, the user hits a deterministic assignment endpoint that returns the same variant based on a stable device signature, ensuring consistency without relying solely on browser cookies.
Interview question
What is the most robust architectural fix when users frequently clear cookies during an A/B test?
- a.Re-assign on each page load and de-duplicate the results during analysis
- b.Extend the first-party cookie expiration to a longer duration
- c.Use IP-based bucketing to avoid dependence on browser storage entirely
- d.Assign variants server-side using a deterministic hash of a stable identifierCorrect
Why? this is the answer
Server-side assignment using a stable identifier ensures the same user always receives the same variant even if browser storage is cleared. Extending cookie expiration is the most tempting distractor because it provides no fallback once the cookie is manually deleted, so re-randomization still contaminates the sample and can trigger Sample Ratio Mismatch.
Just read this? Test yourself on what you have been reading.
Read the original → guessthetest.com
- #ab-testing
- #experimentation
- #architecture
- #cookies
- #server-side
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on ab-testing — each one lists the topics its interview covers.
See open roles