System design
Series · 6 lessons
Processing data: batch and stream
From what the two approaches are, through windows and tools, to a working pipeline you can run for a few dollars.
Start the series- Batch vs stream processing, explained with a simulationBatch collects data and processes it at once, stream handles it as it arrives. Run a live simulation to see what each costs, with a comparison table.
- Batch or stream: which one fits your problem?Pick a real problem to see whether batch, stream or both fits it, and why. With a quick rule for choosing and the Lambda and Kappa architectures.
- Stream windows, event time and late data, explainedAn endless flow cannot be summed once. Watch fixed, sliding and session windows in a live demo, then learn event time, watermarks and how to avoid duplicates.
- Data processing tools: what to choose by language and cloudWhat Spark, Flink, Kafka, Airflow and the services on Cloudflare, AWS, Google Cloud and Azure do. Choose in three layers: industry standard, cloud, language.
- Build a cheap data pipeline with TypeScript on Cloudflare or AWSTake orders, process small batches and total them nightly in one Worker. With a price table by tier, an AWS version and a prompt to hand the job to an AI.
- Build batch pipelines with Spark and stream with Kafka, FlinkThe steps to build a batch pipeline with PySpark and Airflow and a stream pipeline with Kafka and Flink SQL, with samples, safe re-runs and what to learn next.