Design a real-time mobile analytics pipeline

WHAT THIS TESTS: The interviewer wants to see if you understand the fundamental tension between write-heavy ingestion and read-heavy querying in real-time systems. They are looking for awareness of backpressure, exactly-once semantics, and the storage serving layer. Senior candidates should demonstrate that treating ingestion and querying as separate concerns prevents cascading failures and allows independent scaling.

A GOOD ANSWER COVERS: First, the ingestion layer: mobile SDKs should batch events client-side with exponential backoff to avoid overwhelming the pipeline, then emit to a distributed event broker like Apache Kafka or AWS Kinesis with partitioning by user ID or event type to preserve ordering where needed. Second, stream processing: use Apache Flink, Kafka Streams, or Spark Structured Streaming to compute windowed aggregates over tumbling or sliding windows of 1 to 30 seconds, handling late-arriving data with watermarks. Third, the serving layer: sink pre-aggregated metrics into an OLAP database such as ClickHouse, Apache Druid, or Pinot, which can return dashboard queries in under 100 milliseconds even at terabyte scale. Fourth, operational concerns: mention schema evolution with a registry like Confluent Schema Registry, idempotent writes to prevent duplicates during retries, and separate hot and cold paths where raw events land in object storage for replay.

COMMON WRONG ANSWERS: Proposing a single PostgreSQL or MySQL database to handle both ingestion and dashboard queries is a critical red flag because it will collapse under concurrent write and read pressure. Suggesting nightly batch ETL misses the real-time requirement entirely. Another anti-pattern is omitting backpressure handling, which leads to cascading failures when mobile traffic spikes. Failing to distinguish between raw event storage and pre-aggregated metrics also signals shallow experience.

LIKELY FOLLOW-UPS: How do you handle late-arriving events after the aggregation window closes? What is your strategy for exactly-once semantics during consumer rebalances? How would you scale the serving layer if the dashboard needs 10000 queries per second? How do you backfill a new metric without reprocessing the entire stream?

ONE CONCRETE EXAMPLE: A gaming company ingests 2 million events per second from mobile clients. Events land in Kafka partitioned by device ID. Flink consumes the stream, computes 5-second tumbling window aggregates for DAU and session length, and sinks results into ClickHouse. Dashboard queries hit ClickHouse materialized views and return in 50 milliseconds. Raw events are simultaneously written to S3 in Parquet for historical analysis.

Source: Dagster Guides