Batch vs. Stream Processing: When to Process Data
Batch processing is like a nightly report, crunching a full day's data at once. Stream processing is a live feed, handling events as they arrive. Use batch for ETL jobs and stream for real-time fraud detection.
Why it exists
Systems generate vast amounts of data, and we need strategies to analyze it. The core distinction is whether the data is finite and processed in discrete chunks (bounded), or if it's an endless flow processed as it arrives (unbounded). Batch and stream processing are the two fundamental models for handling these scenarios.
The mental model
Think of it as mail delivery. Batch processing is the post office holding all mail and delivering it once a day in a large bundle. It optimizes for throughput. Stream processing is a real-time courier service delivering each package the moment it arrives. It optimizes for latency and data freshness.
How it works
Batch processing systems, like early MapReduce, are designed to read a large, static dataset, perform computations, and write an output. The job runs for a finite time and then terminates. Stream processing systems, like Google's MillWheel, run continuously, ingesting an endless flow of data points. They use concepts like windows (e.g., 'calculate the average over the last 5 minutes') and watermarks to handle the notion of time and completeness in an unending stream.
When to use it
Use batch processing for non-urgent, high-volume tasks where throughput is more important than latency. Examples include end-of-day financial reporting, large-scale data transformations for a data warehouse (ETL), or training ML models on historical data. Use stream processing when low latency is critical, such as for real-time fraud detection, live monitoring dashboards, or updating user session information as clicks happen.
When not to use it
Don't use stream processing for tasks that naturally operate on a complete, finite dataset, as it adds unnecessary complexity. Don't use batch processing when you need immediate insights from live data; the latency will be too high. For example, calculating daily sales figures at midnight is a batch job, but detecting a fraudulent credit card swipe needs to be a stream job.
One canonical example
Before streaming systems, Google computed user session information for Search on a daily basis. This involved a massive batch job that ran overnight, processing the previous day's logs. The move to a streaming model with MillWheel allowed this computation to happen continuously, providing fresher data and more predictable resource usage.
Interview question
A financial institution needs to generate a comprehensive report of all transactions from the previous day for auditing purposes. Which data processing model is most appropriate?
- a.Real-time analytics, to ensure the report reflects the most current financial status.
- b.Batch processing, as it efficiently handles complete, bounded datasets for non-urgent analysis.Correct
- c.Stream processing, because it can continuously ingest and process new transactions as they occur.
- d.Event-driven processing, to react instantly to each transaction for immediate auditing.
Why? this is the answer
Batch processing is ideal for this scenario as it's designed for complete, finite datasets like 'previous day's transactions' and non-urgent tasks such as 'end-of-day financial reporting' or auditing. Stream processing (option C) is for real-time, low-latency needs, not for historical reports on bounded data.
Just read this? Test yourself on what you have been reading.
Read the original → cloud.google.com
- #data engineering
- #data analytics
- #cloud platforms
- #system design
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on data engineering — each one lists the topics its interview covers.
See open roles