Purpose of watermarks in Spark Structured Streaming

WHAT THIS TESTS: Whether you understand the core tension in stateful stream processing, namely that you want to accept late data but cannot keep state forever, and how watermarks resolve it using event time rather than processing time.

A GOOD ANSWER COVERS: In Structured Streaming, aggregations over event-time windows must hold state for each open window. Without a bound that state grows without limit, because in principle a record for any past window could still arrive. A watermark is a moving threshold defined as the maximum event time seen so far minus a configured allowed-lateness interval. Records whose event time is newer than the watermark are still routed into and update their window, so genuinely late data is handled correctly up to the lateness bound. Once the watermark advances past a window's end, Spark considers that window closed: it emits the final result and drops the associated state, so memory stays bounded. Records older than the current watermark are discarded as too late. The key distinction is event time, when the event actually happened, versus processing time, when Spark saw it.

COMMON WRONG ANSWERS: Mixing up event time and processing time, claiming watermarks accept all late data forever, or thinking a watermark drops in-window data. It only drops data beyond the lateness threshold.

LIKELY FOLLOW-UPS: How do you pick the lateness threshold? What output mode interacts with watermarks? What happens to data later than the watermark, is it silently dropped?

ONE CONCRETE EXAMPLE: A job counts events in five-minute windows with a watermark of ten minutes. An event timestamped 12:02 arriving at 12:09 still updates the 12:00 window. Once the watermark passes 12:15, the 12:00 window is finalized and its state freed; a straggler timestamped 12:02 arriving at 12:30 is dropped as too late.