Design a Real-Time Anomaly Detection System for E-commerce Events

What's really being asked

This question evaluates your ability to design an end-to-end, real-time system that solves a specific business problem. It's not just about naming technologies. The interviewer is testing if you understand the core challenge: defining an 'anomaly' in a system with natural, cyclical variance (seasonality). A senior answer moves beyond simple rules to a more robust, data-driven approach, demonstrating an understanding of both data engineering and applied ML.

The full answer

First, an ingestion and streaming layer. Events from the e-commerce application are sent to a scalable streaming platform like Amazon Kinesis Data Streams, which can handle high-volume, unordered events in near real-time.

Second, a transformation and storage strategy. An AWS Lambda function can be triggered by the Kinesis stream to perform lightweight transformations (e.g., parsing, enriching with timestamps). From there, Kinesis Data Firehose batches the data and reliably stores it in an Amazon S3 bucket. This S3 bucket acts as the data lake, providing the historical data needed for model training.

Third, the anomaly detection algorithm. Explicitly state that simple static thresholds are wrong. A drop of 50% at 3 AM might be normal, while a 20% drop at 3 PM is a critical incident. The solution is to use a model that understands seasonality. You can propose a managed service like Amazon Lookout for Metrics, which automatically learns from the historical data in S3 to establish a baseline for any given time and day, then detects significant deviations from it.

Fourth, an alerting and feedback mechanism. When Lookout for Metrics detects an anomaly, it should trigger an Amazon SNS topic. This topic can then fan out notifications to different destinations based on severity: a high-severity alert might go to PagerDuty for the on-call engineer, while a low-severity one posts to a team's Slack channel. The system should also allow engineers to provide feedback on alerts to improve model accuracy over time.

The mistakes people make

Proposing a naive, static threshold algorithm. For example, "alert if the count in a 5-minute window is 20% below the average of the last hour." This is the biggest red flag, as it completely ignores diurnal and weekly traffic patterns and would lead to an unmanageable number of false positives.

Designing a batch-processing system (e.g., a nightly Spark job). The requirement is for real-time detection to catch outages as they happen, not hours later.

Focusing only on one technology without connecting it to the overall architecture. For instance, only talking about Kafka without mentioning how data gets processed, stored, and analyzed.

What usually comes next

How much historical data do you need to train the initial model? (Answer: At least several weeks to capture weekly patterns; several months is better to capture monthly trends.)

What if you can't use a managed service like Lookout for Metrics? How would you build it? (Answer: Mention time-series forecasting models like SARIMA or Prophet, which are designed to handle seasonality, and run them on a compute platform like AWS Fargate or a scheduled Lambda.)

How do you handle a cold start for a new feature with no historical data? (Answer: Start with conservative static thresholds and a human-in-the-loop, collecting data to train a proper model as soon as a baseline is established.)

A concrete example

During Black Friday week, 'add to cart' events might average 50,000 per minute at 2 PM. On a normal Tuesday, the 2 PM average is 8,000. A static threshold would fail completely. An ML model trained on historical data knows to expect 50k events. If the count suddenly drops to 35,000 (a 30% drop), it correctly flags this as a high-severity anomaly. Conversely, if traffic at 4 AM is normally 200 events/minute, a drop to 100 is not a crisis and might not even be flagged, avoiding a false positive that would wake an engineer.