Implement a 10% onboarding tutorial feature flag and track success
Tests lightweight experiment design and metric selection beyond vanity numbers. Strong answers cover user-ID bucketing, conversion tracking, and guardrail metrics like drop-off. Red flag: manual toggles or click-through without downstream outcomes.
What's really being asked
The interviewer wants to see if you understand the difference between shipping a feature and running an experiment. Specifically, they are looking for awareness of randomization, persistence, metric hierarchy, and the ability to detect negative side effects even on a simple ten percent rollout.
The full answer
First, deterministic bucketing. Hash the user ID with a salt modulo one hundred so the same user always sees the same experience and the split is exactly ten percent versus ninety percent. Second, persistent assignment. Store the variant in the user profile or device so that refreshing the app or returning later does not flip the experience. Third, a primary success metric tied to behavior change, such as seven-day activation or time-to-first-key-action, not just tutorial completion. Fourth, guardrail metrics to protect the business, such as overall sign-up abandonment, support ticket volume, or Day-1 retention, because the tutorial could slow users down. Fifth, statistical sanity checks like sample ratio mismatch detection and a pre-defined run time to avoid peeking.
The mistakes people make
Bucketing by request time or IP address, which breaks persistence and randomization. Tracking vanity metrics like page views or button clicks without connecting them to downstream conversion. Using a manual allow-list for ten percent of users, which introduces selection bias and breaks the randomized experiment. Failing to specify a holdout period or stopping the test early the moment numbers look good.
What usually comes next
How would you change the design if the tutorial were server-rendered versus client-side? What would you do if the ten percent cohort showed a five percent lift in activation but a two percent drop in Day-7 retention? How would you handle users who switch devices during onboarding?
A concrete example
Suppose a mobile app wants to test a three-slide tutorial. You would assign new installs to variant A or B at the account-creation step using a hash of the UUID. The flag lives in a remote config service with a default of false. You log an exposure event only when the tutorial is rendered, not at app open, to keep the analysis set clean. Primary metric is the percentage of users who complete the core setup flow within twenty-four hours. Guardrail metrics are app uninstall rate within the first session and average time to complete setup. You run the experiment for two full weeks to cover weekly seasonality and require a p-value below zero point zero five on the primary metric before calling a winner.
Interview question
When rolling out an onboarding tutorial to 10% of new users, which approach best ensures valid experiment results?
- a.Bucket by IP address and track tutorial completion rate as the primary success metric
- b.Assign groups per request and stop early if Day-1 activation improves
- c.Manually select 10% of users and monitor page views over a two-week period
- d.Hash user IDs for persistent assignment and measure 24-hour setup completion while monitoring uninstall ratesCorrect
Why? this is the answer
Hashing user IDs creates deterministic, persistent bucketing that preserves randomization, while 24-hour setup completion is a behavioral primary metric and uninstall rate acts as a guardrail. Option A is tempting but wrong because IP-based bucketing breaks persistence and randomization, and tutorial completion is a vanity metric that ignores downstream outcomes.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
- #experimentation
- #feature-flags
- #growth
- #a/b-testing
- #metrics
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on experimentation — each one lists the topics its interview covers.
See open roles