Architect an A/B test for paid-ad signup flows

What's really being asked

This question tests whether you can design a controlled experiment that spans the boundary between anonymous traffic and authenticated conversion. The interviewer cares about three specific risks: assignment before identity exists, attribution across the signup funnel, and statistical validity at scale. You must show you understand how to bucket users without a database lookup, how to pipe events from a client SDK through a stream processor into queryable aggregates, and how to guard against experiment overlap and selection bias.

The full answer

A good answer hits four things in order. First, assignment at the ad landing page using a deterministic hash of a stable anonymous identifier plus the experiment identifier, producing a bucket number modulo one hundred that maps to variant ranges like zero to forty-nine versus fifty to ninety-nine. This must happen in under one millisecond with no database lookup, using MurmurHash3 or SHA-256, and the variant must be stored in a cookie or device token so it persists through the session. Second, event collection via a client SDK that intercepts the assignment event and every subsequent signup step, batches payloads every five seconds or on page unload, and sends them to an ingestion API backed by Kafka. Third, metric aggregation using a stream processor such as Flink to pre-aggregate counts and sums per hour per variant, storing the results in ClickHouse as tuples of experiment identifier, variant, metric name, bucket hour, value, and user count. Fourth, statistical analysis using two-sample t-tests on signup-rate differences to compute p-values and confidence intervals, plus guardrails like mutual exclusion layers so overlapping experiments on the same surface do not collide, and a small permanent holdout group to measure cumulative lift.

The mistakes people make

Common wrong answers include proposing to assign the variant only after the user begins signup, which severs attribution from the paid ad click and destroys the experiment; suggesting a centralized database to store per-user variant state, which adds latency and cannot scale to millions of assignments per second; ignoring mutual exclusion so users in one signup flow experiment are simultaneously in another, corrupting both metrics; and using simple modulo on user identifier alone without salting by experiment identifier, which forces the same users into the same bucket across every test and introduces dangerous correlation.

What usually comes next

Interviewers often follow up by asking how you would handle cross-device sessions where a user clicks an ad on mobile but converts on desktop; how you would ramp traffic from ten percent to fifty percent without breaking user consistency; how you would detect and mitigate sample ratio mismatch where the hash does not yield a true fifty-fifty split; or how you would compute the required sample size and experiment duration before launch.

A concrete example

Imagine a paid Instagram campaign driving traffic to a new signup flow. When the user taps the ad, the landing page hashes the device advertising identifier plus the experiment identifier, gets bucket sixty-three, and serves variant B. The JavaScript SDK immediately logs an assignment event with the anonymous identifier, experiment identifier, and variant. As the user fills out the form, each step fires an event. On successful signup, a conversion event is fired. All events batch to the API, stream through Kafka, and land in Flink which counts unique converting users per variant per hour. After one week, variant B shows a twelve percent relative lift in signups with a p-value below zero point zero five and a ninety-five percent confidence interval that excludes zero, so the team rolls it out to one hundred percent traffic.