Clickstream architecture for real-time and batch

WHAT THIS TESTS This evaluates whether you can architect a system that simultaneously serves low-latency dashboards and flexible historical analysis, which have conflicting requirements.

A GOOD ANSWER COVERS Start with ingestion: a lightweight collector or SDK on the client sends events to an ingestion endpoint that writes them into a durable, partitioned log such as Apache Kafka or Amazon Kinesis. This decouples producers from consumers and buffers spikes. From the log you fan out into two paths.

The real-time path uses a stream processor like Flink, Spark Structured Streaming, or Kinesis Data Analytics to compute rolling aggregates, sessionization, and metrics, writing results into a low-latency store such as Redis, OpenSearch, or a time-series database that powers live dashboards within seconds.

The batch or analytics path sinks the raw events into a data lake on object storage like S3, partitioned by time, often in a columnar format such as Parquet. Data scientists run ad-hoc, schema-on-read queries with engines like Athena, Presto, or Spark, and can reprocess history. This dual-path design is the lambda architecture; a kappa variant serves both from the stream alone.

WHY TWO PATHS Dashboards need seconds of latency on aggregates; data science needs full-fidelity raw history and flexible queries. No single store optimizes both, so you split responsibilities while sharing one ingestion log.

LIKELY FOLLOW-UPS Lambda versus kappa. How do you handle late or out-of-order events. How do you avoid the small-files problem in the lake. How do you guarantee no event loss.

ONE CONCRETE EXAMPLE A news app sends page-view events to Kinesis. A Flink job updates a per-article view counter in Redis driving a live editor dashboard, while a Firehose delivery stream writes the same raw events as Parquet to S3, where analysts later query reading patterns by region with Athena.