Design an A/B test system for homepage headlines without deployments

WHAT THIS TESTS: This question tests your ability to architect a zero-deploy experimentation pipeline. The interviewer wants to see that you understand separation of concerns between content management, experiment configuration, and application code. Specifically, they are looking for server-side variation selection to avoid layout shift or flash of unstyled content, and a clean event tracking loop that attributes conversions to the correct variant without requiring a new build or release cycle.

A GOOD ANSWER COVERS: First, a headless CMS such as Contentful stores the headline copy outside the codebase, often using a variation container content type that nests two or more entries under a single reference field. Second, an experimentation platform like Optimizely manages the experiment rules, audience segments, and traffic allocation, exposing a datafile that the application consumes. Third, a server-side proxy or middleware fetches the CMS entry, uses the Optimizely SDK to bucket the user deterministically based on a stable user identifier, and resolves the variation container to the specific headline variant before the response reaches the client. Fourth, the frontend renders the resolved headline directly from the server payload, eliminating any client-side swapping. Fifth, conversion events such as sign-up button clicks are tracked via an event SDK call or a webhook that attributes the action back to the experiment and variation identifiers.

COMMON WRONG ANSWERS: A red flag is suggesting that engineers hardcode both headlines behind a feature flag and redeploy to toggle them; this misses the marketer self-service requirement. Another red flag is describing a client-side-only approach where JavaScript swaps the headline after the page loads; this causes cumulative layout shift, flicker, and unreliable attribution. Proposing to store experiment state in a simple database table without an experimentation SDK is also weak because it forces you to rebuild bucketing, audience targeting, and statistical analysis from scratch.

LIKELY FOLLOW-UPS: The interviewer may ask how you handle caching when content changes mid-experiment, how you ensure consistent bucketing across sessions and devices, or how you prevent SEO issues when headlines vary by user. They might also dig into how you would roll out the winning variant to 100 percent traffic without a deploy, or how you would handle multiple concurrent experiments on the same page without interaction effects.

ONE CONCRETE EXAMPLE: Imagine a React application served by a Node proxy. The proxy requests the homepage hero section from Contentful and receives a variation container with two nested entries: headline A says "Start Your Free Trial" and headline B says "Get Started in Seconds." The proxy passes the user's cookie ID into the Optimizely SDK, which returns variant B. The proxy then replaces the variation container with the resolved entry in the JSON response before sending it to the React app. When the user clicks sign up, the frontend calls Optimizely's track method with the event key "signup_completed," attributing the conversion to variant B. The marketing team updates the copy or traffic split in Optimizely and Contentful without ever opening a pull request.

Source: contentful.com