# Data processing tools: what to choose by language and cloud

> What Spark, Flink, Kafka, Airflow and the services on Cloudflare, AWS, Google Cloud and Azure do. Choose in three layers: industry standard, cloud, language.

Canonical URL: https://learn.chabulab.com/system-design/data-processing-technologies. Facts checked: 2026-11-03.

- Kafka moves events, Flink and Spark compute, Airflow schedules. Each has its own role.
- Using the services of the cloud you already run on is usually the fastest and cheapest way to start.
- The order to choose in: SQL first, then the language you already use, and only then learn another one.

Many of the best-known tools are written in Java, but you can use them through SQL, Python or TypeScript without knowing Java. This lesson sorts them into three layers.

> Versions here were checked on 3 November 2026 and change quickly. Check the official pages before you decide.

## Which tools are the industry standard?

These are the names you will meet at most large data companies. Know what they do, even if you will not use them yet.

| Tool | For | Version | Role |
| --- | --- | --- | --- |
| Apache Spark | Batch | 4.3.0 | The most common distributed batch engine. Write SQL, Python or Scala. Offered as Databricks, AWS EMR and Glue, Google Dataproc |
| dbt | Batch | | Transforms data with SQL inside the data warehouse |
| Apache Airflow | Batch | 3.3.2 | Orchestrates pipelines: schedule, order, retries, re-runs by date. Peers: Dagster, Prefect |
| Apache Kafka | Stream | 4.4.0 | A distributed event log. Clients for most languages. Runs on KRaft without ZooKeeper since 4.0 |
| Apache Flink | Stream | 2.3.0 | Strong on state and event time. Mostly used through SQL, Java is not required |
| Spark Structured Streaming | Stream | | Same API as Spark batch. Micro-batch by default |

The price of this group is operations: clusters, configuration, monitoring. If you are not at scale yet, look at layer 2 first.

## What does each cloud offer for the same job?

Using the services of the platform you already run on is usually the fastest and cheapest way to start.

| Role | Cloudflare | AWS | Google Cloud | Azure |
| --- | --- | --- | --- | --- |
| Scheduled runs | Cron Triggers + Workers | EventBridge Scheduler + Lambda | Cloud Scheduler + Cloud Run jobs | Functions (timer), Data Factory |
| Receive and buffer events | Queues, Pipelines | SQS, SNS, EventBridge, Kinesis Data Streams | Pub/Sub | Event Hubs, Service Bus |
| Managed Kafka | Use Queues, or an outside Kafka | Amazon MSK | Managed Service for Apache Kafka | Event Hubs (Kafka interface) |
| Stateful stream processing | Queue consumer + Durable Objects, Pipelines (SQL) | Lambda, Managed Service for Apache Flink | Dataflow | Stream Analytics, Functions |
| Durable workflows | Workflows | Step Functions | Workflows | Durable Functions |
| Store and query for analytics | R2 + Data Catalog + R2 SQL, D1, Analytics Engine | S3 + Athena, S3 Tables, Redshift | BigQuery | Synapse, Fabric |
| Large-scale batch | Pipelines + R2 SQL; outside Spark or DuckDB reading Iceberg on R2 | EMR, Glue | Dataproc, Dataflow, BigQuery | Synapse Spark, Databricks, Data Factory |

Worth noting: Cloudflare bought Arroyo, a stream SQL engine written in Rust, as the base of Pipelines. Pipelines takes events over HTTP or from a Worker, transforms them with SQL and writes to R2 as Iceberg or Parquet.

## What should you choose for the language you use?

The order: SQL first, then your language, and only then learn another one.

### TypeScript and JavaScript

- **On JS infrastructure:** Cloudflare Workers with Queues and Workflows, or AWS Lambda with SQS and Step Functions.
- **Durable workflows:** Cloudflare Workflows, Vercel Workflows, Inngest, Trigger.dev, Upstash Workflow, Temporal.
- **Queues on Redis:** BullMQ for Node.js, when you already have Redis.
- **Kafka from TypeScript:** the Confluent JavaScript client.
- **Analytics:** SQL on D1, Athena, BigQuery or DuckDB.

### Python

- **Batch:** PySpark. DuckDB and Polars for medium data on one machine, very light.
- **Orchestration:** Airflow, Dagster, Prefect.
- **Stream:** PyFlink, the Apache Beam Python SDK, Confluent's Kafka client.
- **Durable workflows:** the Temporal Python SDK.

### Java, Kotlin and Scala

The fullest feature set: Flink, Kafka Streams, Spark itself, Beam Java. A fit when the team already uses the JVM and wants deep control. In return, heavier to start and run.

### Go and Rust

- **Go:** the Temporal Go SDK, franz-go (a Kafka client), Redpanda Connect.
- **Rust:** Arroyo, the stream SQL engine now owned by Cloudflare.

## What are the recent trends?

- **Stream into Iceberg with SQL.** Cloudflare Pipelines with R2 Data Catalog, AWS Kinesis with S3 Tables. Data lands directly in a queryable table.
- **Durable workflows for AI agents.** Retries, waiting for approval and resuming after a failure, with no hand-written queue.
- **Kafka drops ZooKeeper.** Since 4.0 clusters run entirely on KRaft, which is easier to operate.
- **Flink 2.x moves state off the compute machines.** Large state lives in remote storage, so scaling and recovery are faster.
- **Spark gains a Real-Time Mode.** Databricks reports latency of a few hundred milliseconds. That figure comes from the vendor.
- **Batch and stream share one table.** Iceberg, Delta and Paimon allow writing by stream and reading by batch.

## Check what you learned

1. What is Kafka's main role in a stream pipeline?
   Show the answer: Receiving, storing and moving the flow of events. Kafka is a distributed log that connects sources to processors. Flink, Kafka Streams or Spark do the computing. Scheduling is Airflow's job.
2. Your team runs on AWS and needs a simple event queue. What should you look at first?
   Show the answer: SQS, already part of AWS. The services of the platform you already use are usually the fastest and cheapest start. Kafka is for when the numbers call for it.

## Sources

- [Apache Flink: releases](https://flink.apache.org/downloads/)
- [Apache Kafka: release lines (endoflife.date)](https://endoflife.date/apache-kafka)
- [Apache Spark: release lines (endoflife.date)](https://endoflife.date/apache-spark)
- [Apache Airflow: release lines (endoflife.date)](https://endoflife.date/apache-airflow)
- [Cloudflare Pipelines](https://developers.cloudflare.com/pipelines/)
- [Cloudflare acquires Arroyo](https://blog.cloudflare.com/cloudflare-acquires-arroyo-pipelines-streaming-ingestion-beta/)
- [Confluent JavaScript client for Apache Kafka](https://docs.confluent.io/kafka-clients/javascript/current/overview.html)
- [Databricks: Real-Time Mode for Spark Structured Streaming](https://www.databricks.com/blog/announcing-general-availability-real-time-mode-apache-spark-structured-streaming-databricks)