Implement a 10% onboarding tutorial feature flag and track success

WHAT THIS TESTS: The interviewer wants to see if you understand the difference between shipping a feature and running an experiment. Specifically, they are looking for awareness of randomization, persistence, metric hierarchy, and the ability to detect negative side effects even on a simple ten percent rollout.

A GOOD ANSWER COVERS: First, deterministic bucketing. Hash the user ID with a salt modulo one hundred so the same user always sees the same experience and the split is exactly ten percent versus ninety percent. Second, persistent assignment. Store the variant in the user profile or device so that refreshing the app or returning later does not flip the experience. Third, a primary success metric tied to behavior change, such as seven-day activation or time-to-first-key-action, not just tutorial completion. Fourth, guardrail metrics to protect the business, such as overall sign-up abandonment, support ticket volume, or Day-1 retention, because the tutorial could slow users down. Fifth, statistical sanity checks like sample ratio mismatch detection and a pre-defined run time to avoid peeking.

COMMON WRONG ANSWERS: Bucketing by request time or IP address, which breaks persistence and randomization. Tracking vanity metrics like page views or button clicks without connecting them to downstream conversion. Using a manual allow-list for ten percent of users, which introduces selection bias and breaks the randomized experiment. Failing to specify a holdout period or stopping the test early the moment numbers look good.

LIKELY FOLLOW-UPS: How would you change the design if the tutorial were server-rendered versus client-side? What would you do if the ten percent cohort showed a five percent lift in activation but a two percent drop in Day-7 retention? How would you handle users who switch devices during onboarding?

ONE CONCRETE EXAMPLE: Suppose a mobile app wants to test a three-slide tutorial. You would assign new installs to variant A or B at the account-creation step using a hash of the UUID. The flag lives in a remote config service with a default of false. You log an exposure event only when the tutorial is rendered, not at app open, to keep the analysis set clean. Primary metric is the percentage of users who complete the core setup flow within twenty-four hours. Guardrail metrics are app uninstall rate within the first session and average time to complete setup. You run the experiment for two full weeks to cover weekly seasonality and require a p-value below zero point zero five on the primary metric before calling a winner.

Source: Wikipedia: A/B testing