Design a real-time anomaly detection system for 'add to cart' events

What's really being asked

This question tests your ability to design an end-to-end, real-time data analytics system. It's not just about naming services; it's about connecting a critical business problem (a potential outage) to a robust technical solution. The interviewer is evaluating if you can move beyond simple thresholding and account for real-world data complexities like seasonality and noise. They are looking for an understanding of both the data pipeline architecture and the statistical or ML modeling required for accurate, low-false-positive anomaly detection.

The full answer

A strong answer outlines four key stages. First, data ingestion: use a streaming platform like AWS Kinesis or Apache Kafka to collect 'add to cart' events in real-time from application clients or backend services. Second, stream processing and aggregation: use a tool like AWS Lambda, Apache Flink, or Spark Streaming to aggregate raw events into a time series metric, such as 'adds-per-minute'. This creates a clean signal for analysis. Third, the detection algorithm: this is the core. Explicitly reject static thresholds. Instead, propose comparing the current metric to a dynamic baseline that accounts for seasonality. Options include comparing to the same time period last week (e.g., 2:15 PM Tuesday vs last Tuesday), or using an ML-based service like Amazon Lookout for Metrics which learns patterns automatically. Fourth, alerting: when an anomaly is detected, the system must trigger a high-priority alert to an on-call team via a service like PagerDuty, using AWS SNS as the notification hub.

The mistakes people make

The most common mistake is proposing a simple, static threshold like "alert if adds-per-minute drops below 100." This is naive as it ignores normal fluctuations, such as the massive difference in traffic between 3 PM and 3 AM, leading to constant false alarms. Another red flag is designing a batch-based system (e.g., a cron job running every hour) when the requirement is for real-time detection. Finally, a weak answer focuses only on the infrastructure without defining what 'a sudden drop' actually means algorithmically.

What usually comes next

How do you handle a flash sale that causes a massive traffic spike? Does that break your model? How would you backtest your algorithm to tune its sensitivity? What is the expected end-to-end latency of your proposed system, from event to alert? How would you reduce false positives during a holiday like Black Friday where traffic patterns are completely different from normal?

A concrete example

Using AWS, 'add to cart' events are sent to a Kinesis Data Stream. An AWS Lambda function, triggered every 60 seconds, reads events from the stream, calculates the total count for that minute, and sends this data point to Amazon Lookout for Metrics. Lookout, pre-trained on weeks of historical data, has learned the site's normal traffic patterns, including time-of-day and day-of-week seasonality. If the count at 2:15 PM on a Tuesday is 500, but Lookout's model predicted 1,000 with a narrow confidence band, it flags an anomaly. This triggers an SNS topic, which in turn sends a high-priority alert to the SRE team's PagerDuty with context about the deviation.