Design a Real-Time Analytics Pipeline for Mobile Events

What's really being asked

This question tests your understanding of real-time data architectures (like the Kappa architecture), not just traditional batch ETL. The interviewer wants to see if you can select appropriate technologies for each stage of a low-latency pipeline and justify your choices based on trade-offs like throughput, latency, cost, and operational complexity. They are probing for experience beyond simple database writes and looking for an end-to-end vision.

The full answer

A strong answer walks through the pipeline in order, addressing the 'seconds' latency requirement at each step. First, ingestion from millions of clients, using a scalable message queue like Apache Kafka or AWS Kinesis to act as a durable buffer and decouple producers from consumers. Second, stream processing using a framework like Apache Flink or Spark Streaming to perform transformations, filtering, or aggregations in real-time as data flows through. Third, storage in a specialized database, specifically a real-time OLAP datastore like Apache Druid or ClickHouse, which are designed for fast analytical queries over time-series data and high-volume ingestion. Finally, the dashboard layer that queries this OLAP database.

The mistakes people make

A huge red flag is suggesting a traditional batch processing architecture. For example, 'We'll log events to files, then run a nightly job to load them into our data warehouse.' This completely misses the 'within seconds' requirement. Another common mistake is choosing a general-purpose OLTP database like Postgres or MySQL for the final storage. These databases will not scale for the analytical query patterns and high ingest volume required, leading to slow dashboards and query timeouts. Simply naming a technology, like 'We'll use Kafka,' without explaining its role and how it connects to other components is a junior-level response.

What usually comes next

Expect questions about scalability, reliability, and cost. How do you handle backpressure if a downstream component slows down? How do you ensure exactly-once processing semantics to avoid data loss or duplication? How would you handle schema evolution when the mobile app changes its event format? How would you monitor this pipeline for data quality and end-to-end latency? What are the cost implications of your chosen cloud services at 1 million events per second?

A concrete example

For a mobile app generating 100,000 events per second, we'd use a client-side SDK to batch events and send them to a load-balanced endpoint that publishes to a Kafka topic with 50 partitions. A Flink job consumes from Kafka, performing a simple sessionization transformation in a 5-minute tumbling window. The aggregated results are written to Apache Druid, which can ingest millions of events per second and provide P95 query latencies under 2 seconds. The dashboard itself could be a tool like Superset or a custom React app that generates Druid SQL queries.