Skip to content
tezvyn:
Menu

Build a simple A/B test for a headline

Source: interviewEasyHow cards are made

Summary

basic A/B test mechanics.

Key points

assign each visitor a sticky bucket, serve the matching headline variant, log impressions and conversions per variant.

Watch out for

re-randomizing on every load so a user sees both variants.

What's really being asked

It checks that you know the essential client and server pieces of an A/B test and can explain why each one exists rather than just naming a tool.

The full answer

Three parts. First, assignment: when a visitor arrives, randomly bucket them into A or B and persist that choice, typically in a cookie or tied to a user id, so the same person consistently sees the same headline on return visits, which is essential for valid measurement. Second, serving: render the assigned variant. Doing it server-side avoids a flash of the wrong headline, while doing it client-side is simpler but can flicker noticeably. Third, tracking: log an impression event recording the variant shown and a conversion event when the user takes the goal action, sending these to an analytics or events backend. With per-variant impression and conversion counts you compute each variant's conversion rate and run a significance test before declaring a winner.

The mistakes people make

Re-randomizing on every page load, so a user sees A then B and the data is polluted. Not persisting assignment at all. Forgetting to log impressions, which leaves no denominator for the conversion rate. Calling a winner from a small sample without any statistical check.

What usually comes next

How do you ensure sticky assignment survives across sessions and devices. How do you avoid the flicker problem with client-side rendering. What sample size or significance level do you need before deciding. How do you split traffic fifty-fifty reliably and without bias.

A concrete example

A new visitor with no cookie is randomly assigned variant B, stored in a cookie; the server renders headline B; the client logs an impression for B; later the user signs up and a conversion event for B is logged. Aggregating across users, B converts at six percent versus four percent for A, and a significance test confirms whether that lift is real before you roll variant B out to everyone.

Interview question

Why must an A/B test persist each visitor's variant assignment rather than re-randomize on every page load?

  • a.To make the page load faster
  • b.So a user consistently sees one variant, keeping measurement validCorrect
  • c.To reduce server CPU usage
  • d.Because cookies are required by privacy law
Why?

Sticky assignment ensures each user experiences a single variant, so impressions and conversions attribute correctly; re-randomizing mixes exposures and invalidates the comparison. Performance and legal claims are unrelated to stickiness.

Just read this? Test yourself on what you have been reading.

Read the original → optimizely.com

Put your scrolling time to good use

Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.

The iPhone app is on the way

We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.

Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.

Get it on Google PlayiPhone app coming soon

We are hiring for this. Open roles that interview on ab-testing — each one lists the topics its interview covers.

See open roles