Design a real-time mobile analytics pipeline
Tests decoupling high-volume ingestion from low-latency querying. Strong designs use an event broker, a stream processor for windowed aggregates, and an OLAP database for sub-second dashboards.
What's really being asked
The interviewer wants to see if you understand the fundamental tension between write-heavy ingestion and read-heavy querying in real-time systems. They are looking for awareness of backpressure, exactly-once semantics, and the storage serving layer. Senior candidates should demonstrate that treating ingestion and querying as separate concerns prevents cascading failures and allows independent scaling.
The full answer
First, the ingestion layer: mobile SDKs should batch events client-side with exponential backoff to avoid overwhelming the pipeline, then emit to a distributed event broker like Apache Kafka or AWS Kinesis with partitioning by user ID or event type to preserve ordering where needed. Second, stream processing: use Apache Flink, Kafka Streams, or Spark Structured Streaming to compute windowed aggregates over tumbling or sliding windows of 1 to 30 seconds, handling late-arriving data with watermarks. Third, the serving layer: sink pre-aggregated metrics into an OLAP database such as ClickHouse, Apache Druid, or Pinot, which can return dashboard queries in under 100 milliseconds even at terabyte scale. Fourth, operational concerns: mention schema evolution with a registry like Confluent Schema Registry, idempotent writes to prevent duplicates during retries, and separate hot and cold paths where raw events land in object storage for replay.
The mistakes people make
Proposing a single PostgreSQL or MySQL database to handle both ingestion and dashboard queries is a critical red flag because it will collapse under concurrent write and read pressure. Suggesting nightly batch ETL misses the real-time requirement entirely. Another anti-pattern is omitting backpressure handling, which leads to cascading failures when mobile traffic spikes. Failing to distinguish between raw event storage and pre-aggregated metrics also signals shallow experience.
What usually comes next
How do you handle late-arriving events after the aggregation window closes? What is your strategy for exactly-once semantics during consumer rebalances? How would you scale the serving layer if the dashboard needs 10000 queries per second? How do you backfill a new metric without reprocessing the entire stream?
A concrete example
A gaming company ingests 2 million events per second from mobile clients. Events land in Kafka partitioned by device ID. Flink consumes the stream, computes 5-second tumbling window aggregates for DAU and session length, and sinks results into ClickHouse. Dashboard queries hit ClickHouse materialized views and return in 50 milliseconds. Raw events are simultaneously written to S3 in Parquet for historical analysis.
Interview question
In a real-time mobile analytics pipeline using Kafka and Flink, what is the primary architectural benefit of serving dashboards from ClickHouse rather than querying Flink or Kafka directly?
- a.Flink's windowed aggregates are held in transient state that is lost after the window fires, making direct queries unreliable for dashboards
- b.Sinking to ClickHouse removes the need to partition the event broker by user ID since the OLAP layer handles all ordering requirements
- c.It isolates read-heavy dashboard traffic from ingestion and stream processing, allowing each layer to scale independently and preventing cascading failuresCorrect
- d.Kafka and Flink are optimized for sequential stream processing, not for concurrent, random-access dashboard queries at thousands of requests per second
Why? this is the answer
The card emphasizes that treating ingestion and querying as separate concerns prevents cascading failures and allows independent scaling. While Kafka and Flink are not ideal serving layers, the primary architectural benefit is isolating read-heavy query traffic from write-heavy ingestion and processing, not merely query performance or state durability.
Just read this? Test yourself on what you have been reading.
Read the original → dagster.io
- #analytics
- #stream-processing
- #system-design
- #real-time
- #data-pipelines
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on analytics — each one lists the topics its interview covers.
See open roles