Design a real-time personalized notification trigger system

WHAT THIS TESTS: This question tests whether you understand the difference between fire-and-forget messaging and time-bound scheduled delivery at scale. The interviewer wants to see that you can decouple event ingestion from execution, handle backpressure, and design for operability by non-technical teams. They also care about data consistency, exactly-once semantics, and graceful degradation under load.

A GOOD ANSWER COVERS: First, an event ingestion layer such as Kafka or Kinesis that captures user actions with ordering guarantees per user. Second, a distributed delay queue or scheduler like RabbitMQ delayed messages, SQS with visibility timeouts, or a custom time-wheel service that can hold millions of triggers without polling. Third, a rule engine and UI that lets marketers define triggers, audiences, and templates without deploying code, ideally compiling rules into a lightweight DSL or config evaluated by workers. Fourth, idempotent delivery workers that fetch due jobs, resolve user data, render personalized content, and send via push providers while respecting rate limits and opt-outs. Fifth, observability including per-campaign latency metrics, dead-letter queues for failed sends, and circuit breakers for downstream provider outages.

COMMON WRONG ANSWERS: The biggest red flag is proposing a cron job that polls a relational database for rows where scheduled_time is less than now. This creates a hot shard on the time index, misses ordering guarantees, and falls over under high cardinality. Another mistake is skipping idempotency, which leads to duplicate notifications if the worker retries. A third error is ignoring the non-technical requirement by suggesting every rule change requires an engineer to modify code and redeploy.

LIKELY FOLLOW-UPS: The interviewer may ask how you handle a user who returns and completes the action before the 30-minute timer fires. You should answer with a cancellation mechanism, either by removing the scheduled job or by checking state at delivery time. They may also ask about fairness between users or campaigns, which leads to token buckets or per-user rate limiters. Another follow-up is global scale and timezone handling, which requires scheduling in UTC and localizing at render time.

ONE CONCRETE EXAMPLE: Imagine a cart abandonment flow. When a user adds an item, the checkout service emits a CartUpdated event. A stream processor writes a scheduled notification record into a delay queue with a 30-minute TTL. If the user checks out within that window, a CheckoutCompleted event triggers a cancellation request that removes the job from the queue. After 30 minutes, a worker pulls the job, verifies the cart is still abandoned, fetches the user's preferences, renders a localized push, and sends it through Firebase or APNS. If the send fails, the worker retries with exponential backoff up to a limit, then moves the job to a dead-letter queue for manual inspection.