Design a near real-time analytics pipeline for a critical metric

What's really being asked

This question assesses your ability to design for specific data latency requirements. The interviewer is looking for more than a list of technologies; they want to see you reason about the fundamental differences and trade-offs between stream processing (near real-time) and batch processing. They're testing your understanding of concepts like event time vs. processing time, windowing, state management, and the associated operational costs and complexities.

The full answer

A strong answer walks through a logical four-stage architecture. First, ingestion, using a durable message queue like Apache Kafka to handle high-throughput writes and buffer data. Second, processing, using a stream processor like Apache Flink for its low-latency, stateful processing capabilities to aggregate 'orders per minute'. Third, storage, using a specialized OLAP datastore like Apache Druid or ClickHouse, designed for fast queries over time-series data. Fourth, visualization and alerting, using a tool like Grafana to query the datastore and display the metric, with alerts for anomalies. The answer must also contrast this with a batch system (e.g., ETL jobs orchestrated by Airflow loading a data warehouse like Snowflake daily), highlighting the batch system's simplicity and lower cost but unacceptably high latency for this use case.

The mistakes people make

A major red flag is treating this like a standard web application architecture and suggesting relational databases (like Postgres) for storing and querying the high-frequency event stream; they can't handle the write load or query speed. Another mistake is simply listing tools ('I'd use Kafka, Spark, and Druid') without explaining why each component is chosen and how they connect. A junior answer focuses only on the happy path and ignores challenges like handling late-arriving data, ensuring data quality, or managing the state of the stream processor.

What usually comes next

How would you ensure exactly-once processing semantics? How do you handle schema evolution in the event stream? How would you scale the ingestion layer if traffic triples? What if the business now needs analytics with 50ms latency instead of 1-minute latency? How do you monitor the health of this pipeline itself?

A concrete example

For 'orders per minute', our pipeline needs to handle maybe 100,000 events per second at peak. We'd use Kafka for ingestion, as it can easily handle this throughput. Flink would process these events in one-minute tumbling windows. The output—a single data point per minute with the order count—is then written to Druid. Grafana queries Druid every 10 seconds to refresh the dashboard. The end-to-end latency from order event to dashboard update would be around 65-70 seconds, which fits the 'near real-time' requirement. A daily batch job would have a latency of up to 24 hours.