When is streaming better than batch, and what are its infrastructure challenges?

WHAT THIS TESTS: This question probes two distinct senior competencies. First, can you identify a business scenario where latency and actionability matter more than historical completeness? Second, do you understand that streaming is not just batch with smaller files, but a fundamentally different operational paradigm with its own failure modes, consistency guarantees, and cost profile? The interviewer wants to hear you reason about time-to-decision and continuous operational burden.

A GOOD ANSWER COVERS: A strong response starts with a crisp use case where sub-minute or sub-second reaction time creates business value. Fraud detection during payment authorization is a classic example, as is IoT equipment anomaly alerting or real-time ad bidding. You should explicitly contrast this with batch by noting that nightly reconciliation would let the fraud complete or the machine break before detection. Next, you must enumerate infrastructure challenges in four layers. At the ingestion layer, mention handling out-of-order events, duplicate data, and schema evolution without stopping the pipeline. At the processing layer, discuss exactly-once semantics, stateful operations like windowed aggregations, and checkpointing for recovery. At the serving layer, note the need for low-latency stores and the complexity of serving incomplete or late-arriving data. At the operational layer, cite backpressure management during traffic spikes, the difficulty of replay and reprocessing, and the 24/7 on-call burden because there is no nightly downtime window. You should also mention cost, since streaming compute is generally more expensive per record than batch.

COMMON WRONG ANSWERS: A major red flag is describing streaming as batch processing that runs more frequently. Another is picking a use case that does not actually need low latency, such as monthly financial reporting, and forcing streaming onto it. Candidates often list technology names like Kafka or Flink without explaining the specific problems those tools solve. Ignoring backpressure is common; if you do not mention what happens when producers outpace consumers, you signal a lack of production experience. Similarly, omitting state management or exactly-once processing suggests you have only worked with stateless log shipping, not real stream processing.

LIKELY FOLLOW-UPS: Expect the interviewer to ask how you would handle late-arriving data or out-of-order events in a windowed aggregation. They may probe your choice of processing guarantees, asking when at-least-once is acceptable versus when exactly-once is mandatory. You might also be asked to compare cost models, or to sketch how you would migrate an existing batch pipeline to streaming without losing historical correctness. Another favorite follow-up is asking how you would replay a stream to fix a bad deployment without double-writing results.

ONE CONCRETE EXAMPLE: Imagine a ride-share platform detecting fraudulent driver behavior. A batch job that runs hourly would allow a fake driver to complete multiple rides before flagging. A streaming architecture ingests GPS and transaction events into an event bus, applies a stateful session window to detect impossible velocity or collocated drivers, and triggers an account hold within ten seconds. The infrastructure challenges include maintaining driver state across restarts, handling GPS events that arrive out of order, scaling the inference layer during Friday night spikes without dropping alerts, and evolving the fraud model schema without redeploying the entire topology.