Skip to content
ChabuLab Learn

Learn · Processing data: batch and stream · Part 4 of 6

Data processing tools: what to choose by language and cloud

4 minIntermediateFacts checked: Đọc bằng tiếng Việt

Part 4 of 6 · Processing data: batch and stream
  1. Batch vs stream processing, explained with a simulation
  2. Batch or stream: which one fits your problem?
  3. Stream windows, event time and late data, explained
  4. Data processing tools: what to choose by language and cloud
  5. Build a cheap data pipeline with TypeScript on Cloudflare or AWS
  6. Build batch pipelines with Spark and stream with Kafka, Flink

In short

  • Kafka moves events, Flink and Spark compute, Airflow schedules. Each has its own role.
  • Using the services of the cloud you already run on is usually the fastest and cheapest way to start.
  • The order to choose in: SQL first, then the language you already use, and only then learn another one.

Many of the best-known tools are written in Java, but you can use them through SQL, Python or TypeScript without knowing Java. This lesson sorts them into three layers.

Versions here were checked on 3 November 2026 and change quickly. Check the official pages before you decide.

Which tools are the industry standard?

These are the names you will meet at most large data companies. Know what they do, even if you will not use them yet.

Tool For Version Role
Apache Spark Batch 4.3.0 The most common distributed batch engine. Write SQL, Python or Scala. Offered as Databricks, AWS EMR and Glue, Google Dataproc
dbt Batch Transforms data with SQL inside the data warehouse
Apache Airflow Batch 3.3.2 Orchestrates pipelines: schedule, order, retries, re-runs by date. Peers: Dagster, Prefect
Apache Kafka Stream 4.4.0 A distributed event log. Clients for most languages. Runs on KRaft without ZooKeeper since 4.0
Apache Flink Stream 2.3.0 Strong on state and event time. Mostly used through SQL, Java is not required
Spark Structured Streaming Stream Same API as Spark batch. Micro-batch by default

The price of this group is operations: clusters, configuration, monitoring. If you are not at scale yet, look at layer 2 first.

What does each cloud offer for the same job?

Using the services of the platform you already run on is usually the fastest and cheapest way to start.

Role Cloudflare AWS Google Cloud Azure
Scheduled runs Cron Triggers + Workers EventBridge Scheduler + Lambda Cloud Scheduler + Cloud Run jobs Functions (timer), Data Factory
Receive and buffer events Queues, Pipelines SQS, SNS, EventBridge, Kinesis Data Streams Pub/Sub Event Hubs, Service Bus
Managed Kafka Use Queues, or an outside Kafka Amazon MSK Managed Service for Apache Kafka Event Hubs (Kafka interface)
Stateful stream processing Queue consumer + Durable Objects, Pipelines (SQL) Lambda, Managed Service for Apache Flink Dataflow Stream Analytics, Functions
Durable workflows Workflows Step Functions Workflows Durable Functions
Store and query for analytics R2 + Data Catalog + R2 SQL, D1, Analytics Engine S3 + Athena, S3 Tables, Redshift BigQuery Synapse, Fabric
Large-scale batch Pipelines + R2 SQL; outside Spark or DuckDB reading Iceberg on R2 EMR, Glue Dataproc, Dataflow, BigQuery Synapse Spark, Databricks, Data Factory

Worth noting: Cloudflare bought Arroyo, a stream SQL engine written in Rust, as the base of Pipelines. Pipelines takes events over HTTP or from a Worker, transforms them with SQL and writes to R2 as Iceberg or Parquet.

What should you choose for the language you use?

The order: SQL first, then your language, and only then learn another one.

TypeScript and JavaScript

  • On JS infrastructure: Cloudflare Workers with Queues and Workflows, or AWS Lambda with SQS and Step Functions.
  • Durable workflows: Cloudflare Workflows, Vercel Workflows, Inngest, Trigger.dev, Upstash Workflow, Temporal.
  • Queues on Redis: BullMQ for Node.js, when you already have Redis.
  • Kafka from TypeScript: the Confluent JavaScript client.
  • Analytics: SQL on D1, Athena, BigQuery or DuckDB.

Python

  • Batch: PySpark. DuckDB and Polars for medium data on one machine, very light.
  • Orchestration: Airflow, Dagster, Prefect.
  • Stream: PyFlink, the Apache Beam Python SDK, Confluent's Kafka client.
  • Durable workflows: the Temporal Python SDK.

Java, Kotlin and Scala

The fullest feature set: Flink, Kafka Streams, Spark itself, Beam Java. A fit when the team already uses the JVM and wants deep control. In return, heavier to start and run.

Go and Rust

  • Go: the Temporal Go SDK, franz-go (a Kafka client), Redpanda Connect.
  • Rust: Arroyo, the stream SQL engine now owned by Cloudflare.
  • Stream into Iceberg with SQL. Cloudflare Pipelines with R2 Data Catalog, AWS Kinesis with S3 Tables. Data lands directly in a queryable table.
  • Durable workflows for AI agents. Retries, waiting for approval and resuming after a failure, with no hand-written queue.
  • Kafka drops ZooKeeper. Since 4.0 clusters run entirely on KRaft, which is easier to operate.
  • Flink 2.x moves state off the compute machines. Large state lives in remote storage, so scaling and recovery are faster.
  • Spark gains a Real-Time Mode. Databricks reports latency of a few hundred milliseconds. That figure comes from the vendor.
  • Batch and stream share one table. Iceberg, Delta and Paimon allow writing by stream and reading by batch.

Check what you learned

Pick an answer to see why it is right or wrong.

  1. What is Kafka's main role in a stream pipeline?

    Show the answer

    Receiving, storing and moving the flow of events. Kafka is a distributed log that connects sources to processors. Flink, Kafka Streams or Spark do the computing. Scheduling is Airflow's job.

  2. Your team runs on AWS and needs a simple event queue. What should you look at first?

    Show the answer

    SQS, already part of AWS. The services of the platform you already use are usually the fastest and cheapest start. Kafka is for when the numbers call for it.

Sources