Handling late data in streaming windows
event-time stream processing.
use event-time windows with watermarks to bound lateness, allow a grace period before finalizing, and route data later than that to a side output.
What's really being asked
This evaluates whether you grasp event-time semantics and the watermark mechanism that lets streaming systems produce accurate windowed results despite out-of-order arrival.
The full answer
First, aggregate by event time, the timestamp embedded in each record when it occurred, rather than processing time, the moment the engine sees it, because network delays, retries, and mobile offline buffering cause events to arrive out of order. A watermark is the engine's assertion that it believes no events older than a certain event time will still arrive, advancing as data flows. You configure an allowed lateness or watermark delay, for example a few minutes, which keeps each window's state open past its nominal end so events that arrive a bit late still update the correct hourly bucket. When the watermark passes the window end plus the allowed lateness, the window is finalized and its state can be cleared. Events arriving even later than that bound are too old to fold in cheaply, so route them to a side output, dead-letter topic, or a batch correction process instead of silently dropping them. This is a deliberate tradeoff: a longer allowed lateness improves accuracy but increases latency and the state you must retain.
The mistakes people make
Using processing time, which makes results depend on system delays and ignores when events truly happened. Dropping all late data without a side output, losing information. Setting allowed lateness to infinity, which makes state grow unbounded. Confusing watermarks with simple timeouts.
What usually comes next
How do you choose the watermark delay. What is the cost of large state retention. How do you correct already-emitted results.
A concrete example
An hourly active-user count uses event time with a watermark allowing five minutes of lateness. A mobile event delayed three minutes still updates its correct hour, while an event delayed two hours is sent to a side output for a nightly batch reconciliation rather than corrupting the live count.
Interview question
Why aggregate streaming windows on event time with watermarks rather than on processing time?
- a.Processing time is harder to extract from records
- b.Event time removes the need to retain any window state
- c.Watermarks eliminate all late data automatically
- d.Event time plus watermarks places out-of-order records in their true window within a tolerated latenessCorrect
Why? this is the answer
Event-time windowing with watermarks assigns delayed records to the correct window up to an allowed lateness, giving accurate results. Watermarks do not delete late data, and windows still require retained state during the grace period.
Just read this? Test yourself on what you have been reading.
Read the original → docs.databricks.com
- #streaming
- #flink
- #watermarks
- #event-time
- #data-engineering
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on streaming — each one lists the topics its interview covers.
See open roles