Design a near real-time user interaction tracking and analytics system

WHAT THIS TESTS: The interviewer wants to see if you understand the fundamental tension in real-time analytics: high-volume, firehose-style ingestion must coexist with low-latency, high-concurrency querying. They are looking for layered architecture, explicit latency budgets, technology justification, and operational realism like backpressure and failure modes. Senior candidates should demonstrate knowledge of stream processing, OLAP storage, and client-side reliability.

A GOOD ANSWER COVERS: First, the client layer: batch events in a JavaScript or mobile SDK, use beacon API for unload reliability, and include client-generated UUIDs for deduplication. Second, the ingestion backbone: Kafka or AWS Kinesis for durability and partition scaling, with a retention buffer that absorbs traffic spikes. Third, stream processing: Flink, Kafka Streams, or Spark Structured Streaming to enrich events, handle session windows, and deduplicate within a lookback window. Fourth, the analytical store: ClickHouse or Apache Druid for columnar ingestion and sub-second aggregation on billions of rows. Fifth, the serving layer: a query API with Redis caching for hot dashboards and rate limiting to protect the OLAP engine. Finally, map each stage to a latency budget that sums to under 30 seconds, and describe how backpressure flows upstream and how you handle late-arriving data.

COMMON WRONG ANSWERS: A single PostgreSQL or MySQL instance cannot absorb high-cardinality clickstreams and serve concurrent analytical queries without lock contention. Proposing a daily or hourly Spark batch job misses the real-time requirement entirely. Suggesting Elasticsearch as the primary analytics store without acknowledging its limitations on high-cardinality aggregations or write amplification is another red flag. Ignoring client-side data loss, such as not using beacon or retry queues, shows inexperience with browser unload events.

LIKELY FOLLOW-UPS: How would you handle a sudden 10x traffic spike from a viral campaign? How do you guarantee exactly-once semantics across the pipeline? What is your strategy for backfilling historical data after a schema change? How do you balance pre-aggregation in Flink versus raw event storage in ClickHouse? What privacy or GDPR deletion mechanisms do you build into a streaming architecture?

ONE CONCRETE EXAMPLE: Imagine a 50,000 events-per-second clickstream. The browser SDK batches 100 events and flushes via beacon every 5 seconds. Kafka retains 24 hours with 48 partitions. Flink consumes, parses user agents, and assigns session IDs over a 5-minute window. ClickHouse ingests via Kafka engine or materialized views, storing events in a MergeTree table partitioned by day. A Grafana dashboard queries ClickHouse every 10 seconds, but a Redis layer caches the top 20 queries for 5 seconds. End-to-end p99 latency sits at 12 seconds during normal load and 22 seconds during spikes because Kafka backpressure absorbs the burst.