How do you architect a global notification holdback group?

WHAT THIS TESTS: This question evaluates your ability to design a persistent, statistically valid control group for a notification platform without compromising critical user journeys. The interviewer cares about separation of concerns between experimental marketing messages and mandatory transactional messages, sticky user assignment across distributed systems, and data integrity for longitudinal measurement.

A GOOD ANSWER COVERS: First, deterministic sticky bucketing. Hash the user ID with a holdout salt to generate a number between zero and ninety-nine; if it falls below the holdout threshold, persist the assignment in the user profile or an edge cache so every downstream service sees the same decision. Second, namespace separation. The holdout gate should only apply to marketing or experimental notification types. Transactional messages like password resets, security alerts, and billing notifications must live in a separate namespace that bypasses the holdout check entirely. Third, reliability mechanisms. Evaluate the holdout at decision time inside the notification service, not upstream in a batch job. Use circuit breakers so that if the bucketing store is unavailable you default to sending transactional messages and suppress only marketing ones. Emit audit logs for every holdout decision to detect bleed. Fourth, measurement and lifecycle. Define a pre-registered primary metric before starting the holdout. Aggregate results quarterly or annually to measure cumulative lift. Plan for conclusion: when the holdout ends, users must be deterministically rebucketed into normal experiments, which means the holdout salt or flag must be versioned.

COMMON WRONG ANSWERS: Treating all notifications identically and suppressing password resets for holdout users. Using per-request random assignment instead of sticky user-level bucketing, which destroys statistical validity. Storing holdout state only in a client-side cookie where users can clear it. Failing to consider what happens when the holdout concludes, leaving users permanently excluded from experiments. Proposing a holdout larger than five percent without discussing the trade-off in time to statistical significance.

LIKELY FOLLOW-UPS: How do you prevent a new microservice from accidentally sending marketing notifications to holdout users? What happens if you need to emergency-message all users about a security breach? How do you handle users who move across environments or devices? How do you attribute revenue to the holdout when the user journey spans multiple sessions?

ONE CONCRETE EXAMPLE: Suppose you use a feature flag platform. You create a global holdout flag with a five percent traffic allocation. When the notification dispatcher receives a request, it checks the message type. If the type is transactional, it skips the flag evaluation and sends immediately. If the type is marketing, it evaluates the flag using the user ID as the bucketing key. The user sees the default off variation, which maps to a suppressed send. The decision is logged to your data warehouse. At quarter end, you compare revenue per user between the holdout group and users who received winning notification variants, giving you the true incremental value of the platform.