Part 2 of 6 · Processing data: batch and stream
- Batch vs stream processing, explained with a simulation
- Batch or stream: which one fits your problem?
- Stream windows, event time and late data, explained
- Data processing tools: what to choose by language and cloud
- Build a cheap data pipeline with TypeScript on Cloudflare or AWS
- Build batch pipelines with Spark and stream with Kafka, Flink
In short
- Ask first: if the result were an hour late, would anyone lose out? If not, start with batch. If so, you need stream.
- Many systems use both: batch for exact results, stream for fast ones.
- Lambda runs two paths and keeps two copies of the logic. Kappa has one stream path and replays its log to recompute.
The technology comes later. First find out how much delay your problem can take.
Which approach fits your problem?
Pick a problem to see whether to use batch, stream or both, and why.
Nine real problems, each with its acceptable delay, the reasoning and suggested tools.
What is the quick rule for choosing?
Ask: "If the result were an hour late, would anyone lose out?"
- Nobody loses out: start with batch.
- Someone does: you need stream.
- Both are true in different parts of the system: combine them.
For stream, ask next what delay is acceptable. A dashboard refreshed each minute does not need the same technology as blocking fraud within 100 milliseconds.
When you need both, how does the Lambda architecture work?
Lambda runs two paths side by side. The batch path recomputes all of history for an exact result. The speed path handles the newest data for a fast result. A serving layer merges the two.
- Data sourceAll events
- Batch layerSpark, nightly
- Batch viewExact, slow
- ServingMerged for queries
- Data sourceAll events
- Speed layerFlink, continuous
- Realtime viewFast, temporary
- ServingMerged for queries
The drawback: you write and maintain two copies of the same calculation.
How is the Kappa architecture different?
Kappa has one stream path. Every event stays in a long-lived log. To recompute history, you replay the log through a new stream job.
- Event sourceApps, services
- Long-lived logKafka, kept for days
- Stream processorFix logic, replay the log
- ServingDatabase, OLAP
The drawback: the log has to be kept long enough, and the stream engine has to be strong enough to reprocess a lot of data. Many teams now add a lakehouse so batch and stream read one table.
Common mistakes when choosing
- Choosing stream because it sounds modern. If nobody needs the result now, you pay for machines and complexity and get nothing back.
- Not asking what delay is acceptable. "Real time" is not a requirement. A number such as "under 10 seconds" is.
- Building Lambda from day one. Two copies of the logic are costly to keep in step. Add the second path when you have a measured reason.
- Forgetting that batch can run often. A run every 5 minutes is enough for many "near real time" needs.
Check what you learned
Pick an answer to see why it is right or wrong.
You pay 5,000 employees on the 30th of each month. What fits best?
Show the answer
Batch, once attendance is closed. The data is only complete once the period closes, and nobody needs the result within seconds. Batch is cheaper and easier to check and re-run.
Why do product recommendations often use both batch and stream?
Show the answer
The model learns from history, while what just happened needs an instant update. Training a model is throughput work, which suits batch. What the shopper just viewed is a hot signal, which needs stream.
What is the main drawback of the Lambda architecture?
Show the answer
You write and maintain two copies of the same calculation. The batch and speed paths compute the same thing in two pieces of code. Kappa exists to remove one of them.