Build a reusable A/B framework and its copy API

What's really being asked

This question evaluates your ability to design APIs that hide complexity while exposing the right seams. Interviewers want to see that you understand experimentation as a cross-functional system involving product managers, data scientists, and engineers, not just a randomization utility. They are looking for separation of concerns between assignment, rendering, and analytics, plus awareness of frontend-specific failure modes like layout shift and hydration mismatches.

The full answer

First, a declarative surface for developers. Propose something like an Experiment component or useExperiment hook that takes an experiment key, a stable fallback, and a variants object. Second, a bucketing service that is deterministic per user, usually hashing a user ID plus experiment key into a bucket range so the same user always sees the same variant. Third, anti-flicker and loading states. The framework should either block rendering briefly while resolving assignment or server-render the correct variant to avoid a flash of default content. Fourth, automatic impression logging. The framework should fire an exposure event when a variant actually renders, not when the bucketing happens, because analysts need to compare populations who truly saw the treatment. Fifth, configuration outside code. Mention a registry or CMS where PMs define experiment keys, traffic allocation, and variant weights without deploying code.

The mistakes people make

A red flag is suggesting a simple Math.random call inside a component with no persistence or user stickiness, because that breaks the experiment when the user refreshes. Another red flag is conflating assignment and rendering, such as returning raw strings from a utility function and forcing every team to wire up their own UI. Proposing only server-side rendering without explaining how client-side hydration reconciles the variant is also weak. Finally, omitting analytics entirely suggests you do not understand that an experiment is only as good as its data pipeline.

What usually comes next

The interviewer may ask how you handle zero-traffic cold starts or how to enforce mutually exclusive experiments. They might probe performance, asking how many experiments can run on one page without blocking the critical rendering path. Expect questions about targeting, such as segmenting by geography or user tier, and how that interacts with caching at a CDN edge. You may also be asked how the framework behaves when the experiment service is down.

A concrete example

Imagine a headline test on a landing page. The developer writes Experiment name equals hero headline and fallback equals Save Big Today, then passes variants equals control Save Big Today and treatment Get 50 Percent Off. The framework checks a user cookie for a stable ID, hashes it against the experiment key, resolves treatment, and renders the treatment string. If the bucketing service responds slower than 200 milliseconds, the framework renders the fallback immediately to protect page load metrics and logs a timeout. On mount, it sends an exposure event to the analytics queue with experiment key, variant, and timestamp.