Architect email subject line A/B testing for a large user base

What's really being asked

This question evaluates whether you can bridge backend distributed systems and statistical experimental design. Interviewers want to see you understand randomization, idempotency, event tracking, and hypothesis testing in a real-world messaging pipeline rather than treating A/B testing as a frontend-only concern.

The full answer

A strong architecture has four components. First, deterministic bucketing: hash a stable user identifier with a salt to assign each user to a variant permanently for the experiment duration, which prevents users from flipping groups and polluting the results. Second, variant assembly and delivery: the email service should pull the subject line template from an experiment config at send time, ensuring the same user always receives the same variant even if the job retries. Third, telemetry and deduplication: track sends, deliveries, opens, and clicks via an event pipeline that uses at-least-once delivery with idempotent writes so duplicate events do not inflate counts. Fourth, statistical rigor: define the primary metric such as click-through rate rather than open rate alone, calculate required sample size using power analysis before launch, and run a two-proportion z-test or similar hypothesis test only after reaching the predetermined sample to avoid peeking bias.

The mistakes people make

Red flags include re-randomizing users on every send, which destroys statistical independence; using open rate as the sole success metric without accounting for image pre-fetching or privacy pixels that skew measurements; failing to mention deduplication or exactly-once semantics for events; and peeking at results daily without a fixed stopping rule, which dramatically increases false-positive rates. Another weak pattern is suggesting a simple round-robin assignment instead of randomized user bucketing, because that can introduce temporal bias if send times correlate with user engagement.

What usually comes next

Interviewers often ask how you would handle multiple concurrent experiments, which requires orthogonal hashing or layered bucketing to keep experiments independent. They may probe how to detect and mitigate novelty effects or how to weight users who receive multiple emails differently. A common extension is asking how you would adapt the system for multi-armed bandits versus fixed-horizon A/B tests.

A concrete example

Suppose you have ten million users and two subject lines. You hash each user ID modulo one hundred into two buckets of fifty percent each. Your nightly job enqueues sends with embedded variant IDs. An events service consumes delivery and open confirmations from the email provider, writing deduplicated rows into a data warehouse partitioned by experiment and variant. After two weeks you have one million events per arm. You run a two-proportion z-test on click-through rate and declare a winner only if the p-value is below your pre-registered alpha of zero point zero five and the observed effect exceeds your minimum detectable effect of two percent.