Design a near real-time pipeline to monitor orders per minute

What's really being asked

This question evaluates your ability to design a distributed streaming architecture with end-to-end latency in seconds, not hours. Interviewers want to see that you understand the difference between event time and processing time, can select technologies for each layer of the pipeline, and know why a daily batch ETL job cannot simply be run more frequently to achieve the same result. The focus is on fault tolerance, exactly-once semantics, and operational trade-offs.

The full answer

First, ingestion: propose a distributed log like Apache Kafka, Amazon Kinesis, or Google Pub/Sub to buffer raw order events and decouple producers from consumers. Second, stream processing: use Apache Flink, Spark Streaming, or Kafka Streams to compute orders-per-minute with tumbling or sliding windows keyed by region or payment method if needed; mention event-time processing and watermarks to handle out-of-order events. Third, storage: choose an OLAP database optimized for real-time ingestion and fast aggregations such as Apache Druid, ClickHouse, or Apache Pinot rather than a traditional row-oriented RDBMS. Fourth, visualization: Grafana, Apache Superset, or a custom dashboard querying the OLAP layer with refresh intervals under five seconds. Fifth, contrast with batch: batch pipelines collect data over hours, handle late data trivially, and are cheaper to operate at scale but introduce multi-hour latency; real-time pipelines sacrifice cost and complexity for sub-minute freshness and require handling backpressure, stateful recovery, and duplicate suppression.

The mistakes people make

A major red flag is suggesting a cron job that runs a SQL count query every minute against a transactional database; this is polling, not streaming, and will crush the OLTP system under load. Another mistake is ignoring late-arriving data or assuming perfect ordering; senior candidates must mention watermarks or allowed lateness. Proposing to store raw events in S3 and query with Athena for a real-time dashboard shows a fundamental misunderstanding of latency requirements. Similarly, omitting exactly-once semantics or idempotent sinks means you have not thought through failures and replays.

What usually comes next

How would you handle a backpressure scenario when order volume spikes tenfold during a flash sale? How do you reconcile the real-time pipeline with the daily batch warehouse if the two systems show different order counts? What is your strategy for stateful upgrades of the Flink job without losing the current windowed aggregates? How do you guarantee exactly-once delivery from Kafka to the dashboard?

A concrete example

Imagine an e-commerce platform tracking orders per minute during Black Friday. Producers emit order events to a Kafka topic with three partitions. A Flink job consumes the stream, assigns event timestamps, and applies a one-minute tumbling window to count orders per category. The job checkpoints state to S3 every thirty seconds for recoverability. Results sink to ClickHouse, where a Grafana dashboard queries the counts and alerts if electronics orders drop below a threshold for two consecutive minutes. The equivalent batch pipeline would land events in S3 hourly, run an Apache Spark job to aggregate, and load a data warehouse table overnight; the latency difference is twelve hours versus five seconds, but the batch version is simpler to backfill and audit.