How would you architect long-term holdback experiment groups?

WHAT THIS TESTS: Your ability to design infrastructure for causal measurement over quarters rather than days. Interviewers want to see that you treat a holdback as a longitudinal cohort study, not a standard split test. The system must pin users to their assignment indefinitely, compute metrics weeks or months after assignment, and protect the integrity of both the holdback and any overlapping experiments.

A GOOD ANSWER COVERS: Four architectural layers. First, deterministic bucketing: hash the user ID with a fixed experiment salt and store the assignment in a durable identity graph so it survives app reinstalls or device changes. Second, flag isolation: use a separate holdback configuration namespace evaluated before regular experiment flags so a new feature launch cannot accidentally override the holdback. Third, data plumbing: build a delayed ETL that only computes long-term metrics after the full observation window closes, with late-arriving event backfill within a bounded grace period such as 72 hours; use partition pruning on event time to keep query costs predictable. Fourth, analysis rigor: pre-register the primary long-term metric, use clustered standard errors to handle user-level correlation, and apply survival analysis or intent-to-treat models rather than naive t-tests on only surviving users.

COMMON WRONG ANSWERS: Proposing daily re-randomization which destroys causal identification. Storing holdback state only in local device storage so users flip groups after reinstalls. Computing long-term lift by looking only at users still active at month six, which introduces severe survivorship bias. Ignoring that holdbacks burn sample size and can collide with other experiments that also bucket on user ID.

LIKELY FOLLOW-UPS: How do you handle users who were bucketed but never exposed to the feature. What is your plan if the holdback shows negative long-term value but short-term metrics are positive and leadership wants to ship. How do you prevent a holdback from starving a newer experiment of control users. What is the cost of delaying metric computation by 90 days on pipeline storage.

ONE CONCRETE EXAMPLE: Imagine a ride-sharing app tests a new surge-pricing UI. Seven-day retention lifts 12 percent, but economists suspect it trains users to wait longer before booking. You allocate 5 percent of users to a six-month holdback using a deterministic hash on account ID stored in a DynamoDB identity table. The feature flag service checks the holdback table before evaluating any new pricing experiments. A Spark job runs weekly but only seals the 180-day revenue per user metric after the grace period closes. Analysts compare cumulative revenue using a clustered bootstrap and find the holdback cohort actually generated 8 percent more lifetime value, killing the launch.

Source: cxl.com