Part 3 of 6 · Processing data: batch and stream
- Batch vs stream processing, explained with a simulation
- Batch or stream: which one fits your problem?
- Stream windows, event time and late data, explained
- Data processing tools: what to choose by language and cloud
- Build a cheap data pipeline with TypeScript on Cloudflare or AWS
- Build batch pipelines with Spark and stream with Kafka, Flink
In short
- To get a number out of an endless flow, you cut it into time windows.
- Count by event time, when the event really happened, not by when the system received it.
- A watermark decides how long to wait for late data. Exactly-once needs checkpoints and idempotent writes.
Summing a file is easy because a file ends. A flow of events does not, so stream needs ideas that batch never has to think about.
Why does stream need windows?
An endless flow is never "done" being added up. So we cut it into time windows and compute within each one.
The dots below are events arriving over time. The bars are windows, and the number is how many events a window held when it closed. Change the window type to see the same data give different results.
The same 26 events, counted with three kinds of window.
Which kind of window should you use?
| Kind | How it works | Good for |
|---|---|---|
| Fixed (tumbling) | End to end, no overlap | Periodic numbers, such as orders per minute |
| Sliding | Overlapping, by a slide step | Smooth numbers, such as a moving average |
| Session | Closes after enough silence | One visit by one user |
How is event time different from processing time?
Event time is when the event really happened. Processing time is when the system received it. A slow network pulls them apart, so correct results rely on event time. With processing time, running the same data again gives a different answer.
What is a watermark?
A watermark is a point in time before which the system believes all events have arrived. When the watermark passes the end of a window, the window closes and its result goes out.
An event arriving after that is late data and needs its own rule: drop it, write it somewhere to review, or update the result. This is a trade-off: waiting longer gives more complete data, but later results.
How is state kept when a machine dies?
Counters, totals and sessions are state. A stream processor saves them regularly with checkpoints. When a machine dies, it restores the latest checkpoint and re-reads events from the matching position in the log.
How do at-least-once and exactly-once differ?
| Guarantee | Meaning | Consequence |
|---|---|---|
| At-most-once | Delivered once at most | Data can be lost |
| At-least-once | Delivered until confirmed | Nothing lost, duplicates possible |
| Exactly-once | The result is as if handled once | Needs checkpoints and idempotent writes |
The common approach: accept at-least-once delivery and make the write idempotent, for example writing by key so handling an event again does not double the result.
Common mistakes
- Using processing time because it is easy. Results shift with network speed and cannot be reproduced on a re-run.
- Setting the watermark too tight. Too much late data gets dropped. Too loose and results come late. Measure real lateness before choosing.
- Trusting the queue to prevent duplicates. Most only promise at-least-once, so the write still has to be idempotent.
- Having no path for late data. At the very least, count and record it so you know how much you are losing.
Check what you learned
Pick an answer to see why it is right or wrong.
Why does stream need time windows to compute a total?
Show the answer
An endless flow has no end to sum up to, so it has to be cut into pieces. A number needs a boundary. A window is how you put a boundary on a flow that has no end.
A sliding window is 10 seconds long and slides every 5. How many windows does one event belong to?
Show the answer
2 windows. Each window overlaps the next by half its length, so each event falls into two.
An order is created at 10:03:12 but reaches the system at 10:03:40 because of a slow network. Where should a per-minute report put it?
Show the answer
By event time, in minute 10:03. Event time reflects when it really happened, so the result does not change with network speed.