Watermarks: Defining 'Done' in Event Streams

Why it exists

In distributed systems, data rarely arrives in the order it was created. Network latency and clock skew mean an event from 10:01 AM might arrive after an event from 10:02 AM. If you need to calculate a result over a time window, like 'clicks per minute,' how do you know when you've seen all events for that minute? You can't wait forever.

The mental model

A watermark is like a train conductor shouting "All aboard!" before closing the doors. It's a declaration moving through the stream that says, "I am confident no more events older than timestamp T will arrive." This signal of completeness allows the system to stop waiting, close the time window, and finalize its calculation.

How it works

A watermark is a special record in the data stream that contains a timestamp. When an operator (e.g., a window aggregator) receives a watermark with timestamp T, it knows that it will not see any more regular events with a timestamp less than T. It can then trigger computations for all time windows that end before T. Watermark generation strategies typically track the maximum event time seen so far and subtract a fixed 'allowed lateness' period. This creates a heuristic trade-off between waiting for late data and processing results quickly.

When to use it

Use watermarks for any stateful stream processing that relies on event time. This is the foundation for time-windowed aggregations like tumbling windows (e.g., hourly sums), sliding windows (e.g., a 5-minute moving average), and session windows (e.g., grouping user activity with gaps of inactivity). It's what separates true event-time processing from simpler processing-time logic.

When not to use it

Watermarks are unnecessary if you are using processing time, where you only care about when the system observes an event, not when the event occurred. They are also not needed for stateless operations (like a simple map or filter) or in the rare case where your event stream is guaranteed to be perfectly ordered by event time.

One canonical example

A Flink job counts user clicks in 1-minute tumbling windows. Events arrive out of order. When the system generates and processes a watermark for timestamp 12:35:00, it triggers the final calculation for the 12:33:00-12:34:00 window. The system sums all the clicks it has buffered for that window and emits a single result. Any click event with a timestamp in that window that arrives after the watermark has passed is considered late and will be dropped or handled separately.