Spark Structured Streaming: Unify Batch and Stream

WHY IT EXISTS: Early stream processing in Spark used DStreams, an RDD-based API that was separate from the batch DataFrame API. Teams had to maintain two codebases and mental models for what was essentially the same transformation logic. Structured Streaming was built to unify batch and streaming under a single API so that the same joins, aggregations, and windowing logic run on static tables and live streams without rewriting code.

THE MENTAL MODEL: Imagine a stream not as a sequence of messages but as a table that never stops growing. Every new event is simply a new row appended at the bottom. You write a standard SQL or DataFrame query against this table, and the engine continuously updates the result as rows arrive. It is like asking for the sum of a column in a spreadsheet that someone else is still filling in; the answer refreshes automatically each time new data appears.

HOW IT WORKS: The engine executes the query as a series of micro-batches, typically every few seconds. It tracks input offsets in sources such as Kafka or file streams using a checkpoint directory stored on durable storage like S3 or HDFS. For stateful operations such as aggregations or stream-to-stream joins, it maintains keyed state in memory and snapshots it to the checkpoint store. Output modes control what gets written to the sink. Append mode emits only new rows since the last trigger. Update mode emits only rows that changed. Complete mode rewrites the entire result table every trigger. Event-time processing relies on watermarks to bound how long the engine waits for late data.

WHEN TO USE IT: Reach for Structured Streaming when you already run Spark batch workloads and need to reduce latency without adopting a separate stream processor. It fits event-time ETL from Kafka into Delta Lake, real-time monitoring dashboards with tumbling windows, and enriching a live stream with a static dimension table via a broadcast join.

WHEN NOT TO USE IT: Avoid it when you need sub-second end-to-end latency with strict SLAs, because micro-batch overhead usually keeps latency in the seconds to minutes range. It is also a poor fit for complex event processing that requires intricate pattern matching or per-event timers; Flink CEP handles those better. If your source cannot replay data by offset, exactly-once fault tolerance breaks because recovery depends on re-reading input after a crash.

ONE CANONICAL EXAMPLE: An e-commerce company ingests user clickstream events from Kafka into a Structured Streaming job. The job parses JSON payloads with DataFrame operations, groups events into ten-minute event-time tumbling windows, and counts clicks per category. It sinks the results to a Delta table in append mode and sets a fifteen-minute watermark to handle late data. If the driver restarts, the job recovers its window state and Kafka offsets from checkpoint files and resumes without double-counting events.