Purpose of watermarks in Spark Structured Streaming
streaming state management.
a watermark sets a threshold on event-time lateness, lets late data update windows up to that bound, and tells Spark when to finalize and drop old state.
confusing event time with processing time.
What's really being asked
Whether you understand the core tension in stateful stream processing, namely that you want to accept late data but cannot keep state forever, and how watermarks resolve it using event time rather than processing time.
The full answer
In Structured Streaming, aggregations over event-time windows must hold state for each open window. Without a bound that state grows without limit, because in principle a record for any past window could still arrive. A watermark is a moving threshold defined as the maximum event time seen so far minus a configured allowed-lateness interval. Records whose event time is newer than the watermark are still routed into and update their window, so genuinely late data is handled correctly up to the lateness bound. Once the watermark advances past a window's end, Spark considers that window closed: it emits the final result and drops the associated state, so memory stays bounded. Records older than the current watermark are discarded as too late. The key distinction is event time, when the event actually happened, versus processing time, when Spark saw it.
The mistakes people make
Mixing up event time and processing time, claiming watermarks accept all late data forever, or thinking a watermark drops in-window data. It only drops data beyond the lateness threshold.
What usually comes next
How do you pick the lateness threshold? What output mode interacts with watermarks? What happens to data later than the watermark, is it silently dropped?
A concrete example
A job counts events in five-minute windows with a watermark of ten minutes. An event timestamped 12:02 arriving at 12:09 still updates the 12:00 window. Once the watermark passes 12:15, the 12:00 window is finalized and its state freed; a straggler timestamped 12:02 arriving at 12:30 is dropped as too late.
Interview question
Why does a watermark let Spark Structured Streaming keep state bounded for windowed aggregations?
- a.It speeds up the network so data never arrives late
- b.It defines a lateness threshold past which windows are finalized and their state droppedCorrect
- c.It converts event time into processing time
- d.It buffers all late records indefinitely in memory
Why? this is the answer
Once the watermark passes a window's end, Spark finalizes that window and frees its state, bounding memory. It does not change network behavior, convert time semantics, or retain all late data.
Just read this? Test yourself on what you have been reading.
Read the original → spark.apache.org
- #spark
- #structured-streaming
- #watermark
- #event-time
- #state-management
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles