Design a Real-Time Analytics Pipeline for Mobile Events

What's really being asked

This question evaluates your understanding of modern streaming data architectures and the trade-offs involved in designing for low-latency analytics. The interviewer is looking for your ability to differentiate between batch and streaming patterns, select appropriate technologies for each stage of a "hot path" pipeline, and reason about scalability and fault tolerance under high-throughput conditions. It's a test of practical system design, not just theoretical knowledge.

The full answer

A senior-level answer should walk through the pipeline in four distinct stages. First, event collection via a lightweight mobile SDK and an ingestion service (e.g., a load-balanced API gateway) that validates and forwards events. Second, a durable message queue like Apache Kafka or AWS Kinesis to act as a buffer, decouple services, and handle backpressure. Third, a stream processing framework like Apache Flink or Spark Streaming to perform stateless (filtering) and stateful (sessionization, aggregations) transformations in real-time. Fourth, a specialized, low-latency analytical database (OLAP) like Apache Druid or ClickHouse, which are optimized for fast aggregations over time-series data and can power interactive dashboards with sub-second query responses.

The mistakes people make

A major red flag is proposing a batch-oriented architecture. For example, suggesting that mobile events are dumped into S3 and then processed by a nightly Spark job. This completely misses the "within seconds" requirement. Another common mistake is choosing the wrong sink. Using a transactional database like Postgres or MySQL as the final datastore is a poor choice, as they are not optimized for the analytical query patterns required and will not scale. Similarly, while a general-purpose data warehouse like Redshift or BigQuery is good for analytics, it's often not fast enough for the interactive, sub-second queries implied by the prompt.

What usually comes next

Expect questions about specific trade-offs. Why Kafka over Kinesis? (Control vs. managed service). Why Flink over Spark Streaming? (True event-time streaming vs. micro-batching). How would you handle schema evolution for the mobile events? (Schema registry like Confluent Schema Registry). How do you ensure exactly-once processing semantics? (Idempotent writes, transactional sinks). How would you monitor this pipeline for data quality and latency? (Metrics on queue depth, processing lag, end-to-end latency).

A concrete example

A mobile app sends a "button_click" event via HTTPS to an AWS API Gateway endpoint backed by Lambda functions. The Lambda validates the event and pushes it into an AWS Kinesis stream. An Apache Flink job running on Kubernetes reads from Kinesis, enriches the event with user geo-location, and performs a 1-minute tumbling window count of clicks per feature. The aggregated results are written to an Apache Druid cluster. A Grafana dashboard queries Druid directly, showing the number of clicks per feature, updated every few seconds. The end-to-end latency from click to dashboard visibility is typically under 5 seconds.