Clickstream architecture for real-time and batch

What's really being asked

This evaluates whether you can architect a system that simultaneously serves low-latency dashboards and flexible historical analysis, which have conflicting requirements.

The full answer

Start with ingestion: a lightweight collector or SDK on the client sends events to an ingestion endpoint that writes them into a durable, partitioned log such as Apache Kafka or Amazon Kinesis. This decouples producers from consumers and buffers spikes. From the log you fan out into two paths.

The real-time path uses a stream processor like Flink, Spark Structured Streaming, or Kinesis Data Analytics to compute rolling aggregates, sessionization, and metrics, writing results into a low-latency store such as Redis, OpenSearch, or a time-series database that powers live dashboards within seconds.

The batch or analytics path sinks the raw events into a data lake on object storage like S3, partitioned by time, often in a columnar format such as Parquet. Data scientists run ad-hoc, schema-on-read queries with engines like Athena, Presto, or Spark, and can reprocess history. This dual-path design is the lambda architecture; a kappa variant serves both from the stream alone.

Why two paths

Dashboards need seconds of latency on aggregates; data science needs full-fidelity raw history and flexible queries. No single store optimizes both, so you split responsibilities while sharing one ingestion log.

What usually comes next

Lambda versus kappa. How do you handle late or out-of-order events. How do you avoid the small-files problem in the lake. How do you guarantee no event loss.

A concrete example

A news app sends page-view events to Kinesis. A Flink job updates a per-article view counter in Redis driving a live editor dashboard, while a Firehose delivery stream writes the same raw events as Parquet to S3, where analysts later query reading patterns by region with Athena.