How would you handle late-arriving data in a streaming analytics pipeline?

WHAT THIS TESTS: This question tests whether you understand the difference between event time and processing time and can design for correctness rather than just throughput. Interviewers want to see that you know unbounded streams require explicit bounds on lateness and that you can reason about state accumulation, exactly-once semantics, and observable side effects in sinks.

A GOOD ANSWER COVERS: First, shift from processing time to event time by assigning timestamps from the payload and windowing by event time. Second, define watermarks as the heuristic that tracks event-time progress; set an allowed lateness bound such as ten minutes or one hour based on the source SLAs so the system does not wait forever. Third, use triggers to emit early speculative results and a final result when the watermark passes the window end plus allowed lateness. Fourth, decide what to do with data that arrives after the allowed lateness: drop it, send it to a dead-letter queue, or reprocess it in a separate backfill pipeline. Fifth, guarantee correctness in the sink by making writes idempotent using deterministic window identifiers, or by using two-phase commits so partial failures do not double count events.

COMMON WRONG ANSWERS: A major red flag is relying on processing-time windowing or micro-batch intervals like thirty seconds and assuming that covers late data. Another is suggesting unbounded state by keeping all windows open indefinitely; this leads to memory pressure and GC issues. Saying you will just use a buffer without defining eviction policy or watermark semantics is also weak. Finally, ignoring the sink and assuming the stream processor alone guarantees end-to-end correctness shows a lack of systems thinking.

LIKELY FOLLOW-UPS: The interviewer may ask how you choose the allowed lateness bound in practice; answer by looking at the P99 or P99.9 of observed skew in your source. They may ask what happens if a watermark is too aggressive and emits before late data arrives; explain that you would need to retract previous results or accept temporary inaccuracy. They might also ask about backfilling historical corrections; describe running the same pipeline over bounded historical input to overwrite prior aggregates.

ONE CONCRETE EXAMPLE: Suppose you are counting clicks per minute. You assign event timestamps from the click log. You set fixed one-minute windows, a watermark lag of thirty seconds, and allowed lateness of five minutes. A click with event time 12:00:45 that arrives at 12:06:00 is late but within the bound, so it updates the 12:00 window and triggers a refined pane. A click arriving at 12:08:00 is beyond allowed lateness and routes to a dead-letter table for daily backfill. The sink uses the window start and a deterministic hash as an idempotency key so reprocessing never duplicates counts.