Handling Late-Arriving Data in a Streaming Pipeline

WHAT THIS TESTS: This question probes your understanding of stateful stream processing and the core challenge of reconciling event time (when an event occurred) with processing time (when the system sees it). The interviewer wants to see if you can articulate a robust strategy for handling data that arrives out of order, which is inevitable in distributed systems. They are testing your knowledge of the three key mechanisms: watermarks, windowing, and triggers.

A GOOD ANSWER COVERS: A senior-level answer outlines a three-part strategy. First, distinguish between event time (from the event payload) and processing time. All analytics should be based on event time. Second, introduce watermarks. A watermark is a heuristic that signals when the system believes all data for a certain event time has arrived; it's the system's notion of "completeness". Third, explain windowing, specifically using fixed or sliding windows based on event time to group related events. Finally, describe the trigger mechanism. The default trigger fires when the watermark passes the end of a window. To handle late data, you must configure the trigger with an "allowed lateness" period (e.g., 6 hours). This keeps the window's state active, allowing it to incorporate late events and emit updated, corrected results. Data arriving after the allowed lateness can be sent to a dead-letter queue.

COMMON WRONG ANSWERS: A major red flag is ignoring event time entirely and processing events as they arrive (in processing time). This leads to incorrect aggregations whenever events are out of order. Another weak answer is suggesting periodic, large-scale batch reprocessing of the raw data. While this can achieve correctness, it's inefficient and negates the low-latency benefits of a streaming architecture. A candidate who can't explain what a watermark is or how it relates to windowing is not demonstrating senior-level knowledge. Simply naming a tool like Flink or Beam without explaining the underlying mechanisms is also a poor response.

LIKELY FOLLOW-UPS: How do you choose the "allowed lateness" duration? (It's a business trade-off between accuracy, latency, and cost. Analyze historical data for event delay distribution, e.g., 99.9% of events arrive within 12 hours). What happens to data that arrives after the allowed lateness period? (It's typically dropped or routed to a separate location for manual inspection or a separate batch correction pipeline). How do watermarks handle partitions with idle sources? (An idle partition can stall the global watermark. Solutions involve idle source detection or using processing time advancements after a timeout to move the watermark forward).

ONE CONCRETE EXAMPLE: Imagine counting unique user clicks per minute. We use a 1-minute fixed window based on event timestamps. Our watermark is configured to be the oldest event time seen minus a 30-second buffer. We set an allowed lateness of 5 minutes. An event from 10:01:15 arrives at 10:01:20. It's placed in the [10:01:00, 10:02:00) window. At 10:02:30, the watermark passes 10:02:00, causing the window to fire and emit a preliminary count. At 10:04:00, a late event from 10:01:45 arrives. Because we are within the 5-minute allowed lateness period, the window re-opens, incorporates the new event, and the trigger fires again, emitting a corrected count for that window. An event from 10:01:50 arriving at 10:08:00 would be discarded.