Design a system to A/B test headlines for a single article URL

What's really being asked

This question evaluates your ability to design a controlled experiment system rather than simply toggling a feature flag. Interviewers want to see that you understand randomization with user stickiness, separation of concerns between content storage and experiment configuration, and the difference between counting events versus computing reliable metrics.

The full answer

First, deterministic bucketing. Hash the user ID concatenated with the experiment key to assign a user to a variant. This ensures the same user always sees the same headline across sessions without needing to store per-user assignment rows at scale. Second, data modeling. Keep the article table unchanged. Create an experiments table with fields like experiment_id, article_id, start_time, and status, plus a variants table with variant_id, experiment_id, headline_text, and traffic_allocation. Third, event tracking. When the list page renders, emit an impression event containing timestamp, user_id, experiment_id, and variant_id. When the user clicks through, emit a click event with the same dimensions. Use an analytics warehouse or structured event stream rather than updating a counter in place. Fourth, metric computation. CTR equals clicks divided by impressions per variant. Compute this in batch or with a streaming aggregator, and require statistical significance, typically a p-value below 0.05 or a confidence interval that does not cross zero, before selecting a winner.

The mistakes people make

A red flag is client-side randomization that re-rolls on every page view, which splits a user's behavior across variants and corrupts the data. Another mistake is storing the headline directly on the article record and creating a new article row for each variant, which pollutes the content management system and breaks the single URL requirement. A third error is pre-aggregating CTR in a real-time counter without retaining event-level rows, making it impossible to debug data quality or compute confidence intervals.

What usually comes next

Interviewers may ask how you would handle anonymous users without stable IDs, how to prevent flash-of-wrong-content in server-side rendering, or how to run multiple concurrent experiments on the same page without interaction effects. They might also ask how you would allocate traffic unevenly, such as a ninety-ten split, or how to automatically ramp the winning variant to one hundred percent.

A concrete example

Suppose article slug summer-trends has two headlines. The experiment table links summer-trends to experiment_id exp_42. The variants table holds Control: Top Summer Trends and Variant B: The Only Summer Trends That Matter. A user with ID 12345 hashes to bucket seven, which maps to Variant B. The list page API returns the Variant B headline and logs an impression. If the user clicks, a click event fires. After ten thousand impressions per variant, Control shows a four percent CTR and Variant B shows a five point two percent CTR with a p-value of zero point zero one. You declare Variant B the winner and update the article title.