Handling late data in streaming windows

WHAT THIS TESTS This evaluates whether you grasp event-time semantics and the watermark mechanism that lets streaming systems produce accurate windowed results despite out-of-order arrival.

A GOOD ANSWER COVERS First, aggregate by event time, the timestamp embedded in each record when it occurred, rather than processing time, the moment the engine sees it, because network delays, retries, and mobile offline buffering cause events to arrive out of order. A watermark is the engine's assertion that it believes no events older than a certain event time will still arrive, advancing as data flows. You configure an allowed lateness or watermark delay, for example a few minutes, which keeps each window's state open past its nominal end so events that arrive a bit late still update the correct hourly bucket. When the watermark passes the window end plus the allowed lateness, the window is finalized and its state can be cleared. Events arriving even later than that bound are too old to fold in cheaply, so route them to a side output, dead-letter topic, or a batch correction process instead of silently dropping them. This is a deliberate tradeoff: a longer allowed lateness improves accuracy but increases latency and the state you must retain.

COMMON WRONG ANSWERS Using processing time, which makes results depend on system delays and ignores when events truly happened. Dropping all late data without a side output, losing information. Setting allowed lateness to infinity, which makes state grow unbounded. Confusing watermarks with simple timeouts.

LIKELY FOLLOW-UPS How do you choose the watermark delay. What is the cost of large state retention. How do you correct already-emitted results.

ONE CONCRETE EXAMPLE An hourly active-user count uses event time with a watermark allowing five minutes of lateness. A mobile event delayed three minutes still updates its correct hour, while an event delayed two hours is sent to a side output for a nightly batch reconciliation rather than corrupting the live count.