How do you technically implement an A/B test for onboarding flows?

WHAT THIS TESTS: Whether you can build a statistically valid experiment pipeline rather than just toggling a feature flag. Interviewers care about assignment integrity, data hygiene, and the ability to separate correlation from causation when measuring activation.

A GOOD ANSWER COVERS: Four layers in order. First, deterministic bucketing. Hash the user_id with the experiment key to produce a modulo split, or use a feature-flag service like LaunchDarkly or an in-house variant assignment service. This guarantees the same user always lands in the same bucket even across devices. Second, persistence. Store the assigned variant in the user profile database or a durable user-scoped store, not just a cookie or localStorage, so the assignment survives logout or cross-device sessions. Third, an exposure event. Fire an experiment_exposed event before the variant renders, logging user_id, variant_id, experiment_key, and timestamp. This creates an intent-to-treat cohort and prevents survivorship bias from only tracking users who completed the flow. Fourth, analytics schema. Use a single event stream where every downstream action, onboarding step completion, and activation milestone carries the variant_id as a property. Keep the activation metric predefined and binary, such as completed_profile plus used_core_feature within seven days, rather than a vague engagement score.

COMMON WRONG ANSWERS: Randomizing on every page load or session start, which contaminates the experiment and breaks the user experience. Relying solely on client-side bucketing without server-side validation, which allows users to spoof variants. Tracking only click-through rates or page views instead of a concrete activation event. Failing to log an exposure event, which makes it impossible to distinguish users who saw the flow from those who dropped off before rendering. Using different analytics tables or schemas for each variant, which invites join errors and inconsistent tracking.

LIKELY FOLLOW-UPS: How would you detect sample ratio mismatch? How do you handle users who switch devices mid-onboarding? What if one variant has a bug that crashes the app for five percent of users? How long do you run the test, and what statistical test do you use for the activation rate? How do you prevent the same user from being enrolled in multiple overlapping onboarding experiments?

ONE CONCRETE EXAMPLE: Suppose you have one million daily active users and a 50-50 split. You hash each user_id concatenated with the string onboarding_v2 and take modulo 100. Users 0-49 see variant A and 50-99 see variant B. When the user starts the app, your API returns profile.variant equals B alongside the rest of the user object. The client immediately fires an onboarding_exposed event with variant B. Every subsequent onboarding_step_completed and the final activation event, which is defined as invited_team_member within 48 hours, carries that same variant_id. Your analyst queries the exposure table joined to the activation table by user_id, counts activated users per variant, and runs a chi-squared or z-test on the binary outcome.

Source: optimizely.com