How do you handle late-arriving data in a streaming pipeline?

WHAT THIS TESTS: This question probes your understanding of the core challenge in stream processing: reconciling the time an event occurred (event time) with the time the system observes it (processing time). It tests knowledge of standard patterns like watermarks, windowing, and triggers, which are fundamental in frameworks like Apache Beam or Flink. The interviewer wants to see if you can reason about the trade-offs between correctness, latency, and cost.

A GOOD ANSWER COVERS: A strong answer addresses four points in order. First, clarify the distinction between event time (when the event happened on the client) and processing time (when the pipeline sees it), stating that correctness requires processing based on event time. Second, describe using event-time windowing to group data, for example, creating 5-minute windows based on the event timestamp attached to each record. Third, introduce the concept of watermarks. Explain that a watermark is a heuristic timestamp that advances, signaling that the system believes all data for a period has arrived. The default behavior is to close a window when the watermark passes its end. Fourth, explain how to handle data that arrives after the watermark has passed. You would configure "allowed lateness" (e.g., withAllowedLateness in Beam). This keeps the window's state alive for a specified duration (e.g., 1 hour), and you would use triggers to emit corrected results when late data arrives within that period.

COMMON WRONG ANSWERS: A frequent mistake is to ignore event time entirely and only process based on arrival time. This leads to incorrect analytics, like a sale from 11:58 PM on Friday being counted in Saturday's totals if it arrives at 12:01 AM. Another red flag is suggesting only a batch-based backfill process. While sometimes necessary for extremely late data, modern streaming engines can handle significant lateness in-stream, which is more efficient. Dropping all late data is too simplistic and sacrifices correctness. Finally, confusing a data-driven watermark with a static wall-clock timer is a conceptual error.

LIKELY FOLLOW-UPS: Expect questions like, "How do you decide the duration for 'allowed lateness'?" Your answer should focus on the trade-off between correctness and cost, analyzing historical data delay distribution versus state storage costs. Another follow-up is, "What happens to data that arrives after the allowed lateness period?" The correct answer is to send it to a dead-letter queue for manual inspection or separate batch processing. Finally, be prepared for, "How do downstream systems handle corrected results?" They must support updates, for example, by using an upsert operation in the sink database rather than a simple append.

ONE CONCRETE EXAMPLE: Imagine counting unique users per minute. A window for 12:00-12:01 PM is defined. The watermark passes 12:01 PM, and the system emits a result: "1500 unique users." We set an allowed lateness of 1 hour. At 12:05 PM, an event with a timestamp of 12:00:30 PM arrives. Because it's within the allowed lateness period, the system processes it, re-calculates the count for the 12:00-12:01 PM window, and emits an updated result: "1501 unique users." A downstream database would see this as an UPDATE to the row for the 12:00 window. Data arriving at 1:02 PM for that window would be dropped.