Describe how you'd implement an A/B test for a landing page headline

WHAT THIS TESTS: This question assesses whether you treat experimentation as a full-stack data integrity problem rather than a front-end cosmetic task. Interviewers at the senior level want to see that you understand randomization, attribution, performance, and statistical validity, even when the prompt only asks about an H1 tag.

A GOOD ANSWER COVERS: First, deterministic bucketing. You need a rule that assigns a user to variant A or B consistently across sessions, usually by hashing a user ID or cookie and splitting by percentage. Second, anti-flicker rendering. The headline should be rendered server-side or the page should be hidden until the variant is applied, because swapping text after paint creates layout shift and invalid exposure data. Third, event instrumentation. You must fire an exposure event when the user sees the variant and a separate conversion event when they sign up, with both payloads containing the experiment name and variant ID so analysts can join them. Fourth, success metrics and sample size. You should define the primary metric up front and calculate how many visitors or conversions you need to reach statistical significance before the test begins. Fifth, SEO and performance guardrails. Search engine crawlers should typically receive the control version, and the experiment should not regress Core Web Vitals.

COMMON WRONG ANSWERS: Relying on a client-side script that waits for the DOM to load and then swaps the headline text. This causes a visible flash of original content, biases the data because users may convert before the swap completes, and breaks attribution if the analytics event fires too early. Another red flag is suggesting that users get a new variant on every refresh, which destroys session consistency and makes conversion tracking meaningless. Declaring a winner after a few hours or a handful of sign-ups without mentioning p-values, confidence intervals, or minimum detectable effect also signals weak analytical rigor.

LIKELY FOLLOW-UPS: How would you handle this if the headline change needed to be indexed by search engines? What if the sign-up happens on a different subdomain or in a mobile app days later? How do you prevent stakeholders from peeking at early results and stopping the test as soon as one variant temporarily leads? These probe whether you understand cross-platform attribution, SEO implications, and experiment governance.

ONE CONCRETE EXAMPLE: Imagine a React landing page served by Next.js. The experiment assignment happens at the edge or during server-side rendering so the HTML arrives with the correct H1 already in place. The client fires an exposure event to Segment or Amplitude once per session. The sign-up button triggers a conversion event carrying the experiment key and variant name. The team agrees to run until at least one thousand conversions per variant or two full business weeks, whichever comes last, and uses a chi-squared test or a sequential testing engine to evaluate results.

Source: optimizely.com