# Build a cheap data pipeline with TypeScript on Cloudflare or AWS > Take orders, process small batches and total them nightly in one Worker. With a price table by tier, an AWS version and a prompt to hand the job to an AI. Canonical URL: https://learn.chabulab.com/system-design/build-cheap-data-pipelines-typescript. Facts checked: 2026-11-03. - Start at the lightest tier: serverless, a queue and cron. Move up only when measured numbers call for it. - One Worker can do all three jobs: take events, process small batches and run a scheduled batch. - To get good work from an AI, give it the event contract, the idempotency rule, a cost ceiling and a way to check itself. When an AI writes and runs code for you, what matters most is a system with few parts that can be tried right away, checked with SQL and retried safely. So start at the lightest tier, and move up only when measured numbers call for it. ## Which tier should you start at, and what does it cost? | Tier | When | Cloudflare | AWS | Rough cost | | --- | --- | --- | --- | --- | | 1. Serverless + queue + cron | Starting out, each event is a small request | Worker, Queues, D1, Cron Triggers | Lambda, SQS, DynamoDB, EventBridge Scheduler | Workers Paid $5/month with 10 million requests. Queues: 1 million operations a month, then $0.40/million. Lambda and SQS each have 1 million free requests a month | | 2. Stream into a data lake, query with SQL | Analytics on large data | Pipelines, R2 Data Catalog, R2 SQL | Kinesis or Firehose, S3, Athena | Pipelines: first 50 GB/month, then $0.04/GB transform and $0.06/GB Parquet sink. Kinesis on-demand: $0.04/stream-hour, about $29/month, plus per-GB fees | | 3. Kafka + Flink or Spark | Large state, strict exactly-once, hundreds of thousands of events a second | An outside Kafka | MSK, Managed Flink, Databricks | Depends on scale, so work it out from real numbers | A message on Cloudflare Queues usually costs 3 operations: write, read, delete. AWS prices are for US East. Pipelines is in beta, so its pricing may change. ## How do you build one Worker that does all three jobs? The problem: a shop takes orders, wants to store them in small batches and total revenue each night. One Worker does all three. HTTP → Queue → Worker queue() → D1 → Nightly cron > Run locally on 3 November 2026 with wrangler 4.151.3: sending an order twice did not double it, a malformed order got a 400, and cron computed the right total for the day before. Not yet deployed to a real account. ```jsonc // wrangler.jsonc { "name": "orders-pipeline", "main": "src/index.ts", "compatibility_date": "2026-10-01", "queues": { "producers": [{ "queue": "orders", "binding": "ORDERS" }], "consumers": [{ "queue": "orders", "max_batch_size": 50, "max_batch_timeout": 5 }] }, "d1_databases": [ { "binding": "DB", "database_name": "orders-db", "database_id": "" } ], // Cron is in UTC: 19:00 UTC is 02:00 in Vietnam "triggers": { "crons": ["0 19 * * *"] } } ``` ```sql -- schema.sql CREATE TABLE orders ( order_id TEXT PRIMARY KEY, shop_id TEXT NOT NULL, amount INTEGER NOT NULL, ts INTEGER NOT NULL ); CREATE TABLE daily_revenue ( day TEXT, shop_id TEXT, so_don INTEGER, doanh_thu INTEGER, PRIMARY KEY (day, shop_id) ); ``` ```ts // src/index.ts import { z } from "zod"; // The event contract: everything is checked at the door const OrderEvent = z.object({ orderId: z.string().min(1), shopId: z.string().min(1), amount: z.number().int().nonnegative(), ts: z.number().int(), // event time, milliseconds }); type OrderEvent = z.infer; interface Env { ORDERS: Queue; DB: D1Database; } export default { // 1) Take the event over HTTP, put it on the Queue and answer at once async fetch(req: Request, env: Env): Promise { const parsed = OrderEvent.safeParse(await req.json()); if (!parsed.success) return new Response("invalid event", { status: 400 }); await env.ORDERS.send(parsed.data); return new Response("ok", { status: 202 }); }, // 2) Micro-batch stream: the Queue collects up to 50 messages or 5 seconds, then calls once async queue(batch: MessageBatch, env: Env): Promise { const stmt = env.DB.prepare( `INSERT INTO orders (order_id, shop_id, amount, ts) VALUES (?1, ?2, ?3, ?4) ON CONFLICT(order_id) DO NOTHING`, // idempotent: handling it again does not double it ); await env.DB.batch( batch.messages.map((m) => stmt.bind(m.body.orderId, m.body.shopId, m.body.amount, m.body.ts)), ); batch.ackAll(); }, // 3) Batch: a Cron Trigger runs each night and computes yesterday's revenue async scheduled(_controller: ScheduledController, env: Env): Promise { await env.DB.prepare( `INSERT OR REPLACE INTO daily_revenue (day, shop_id, so_don, doanh_thu) SELECT date(ts / 1000, 'unixepoch'), shop_id, COUNT(*), SUM(amount) FROM orders WHERE date(ts / 1000, 'unixepoch') = date('now', '-1 day') GROUP BY shop_id`, ).run(); }, } satisfies ExportedHandler; ``` Run these in order: ```bash npm install zod npx wrangler d1 create orders-db npx wrangler d1 execute orders-db --local --file=schema.sql npx wrangler queues create orders npx wrangler dev --test-scheduled ``` Check it with: ```bash TS=$(( ($(date +%s) - 86400) * 1000 )) curl -X POST localhost:8787 -d "{\"orderId\":\"A1\",\"shopId\":\"s7\",\"amount\":100,\"ts\":$TS}" curl -X POST localhost:8787 -d "{\"orderId\":\"A1\",\"shopId\":\"s7\",\"amount\":100,\"ts\":$TS}" curl "localhost:8787/__scheduled?cron=0+19+*+*+*" npx wrangler d1 execute orders-db --local --command "SELECT * FROM daily_revenue" ``` The `daily_revenue` table should show `so_don` as 1 even though the order was sent twice. When it works, change `--local` to `--remote` and run `npx wrangler deploy`. ## How does it look on AWS? API Gateway → SQS → Lambda by batch → DynamoDB Turn on `ReportBatchItemFailures` on the event source mapping so only failed messages are retried. For the nightly batch, EventBridge Scheduler calls Lambda or Step Functions. This sample has not been run on a real AWS account. ```ts // handler.ts import type { SQSBatchResponse, SQSHandler } from "aws-lambda"; import { DynamoDBClient } from "@aws-sdk/client-dynamodb"; import { DynamoDBDocumentClient, PutCommand } from "@aws-sdk/lib-dynamodb"; import { z } from "zod"; const OrderEvent = z.object({ orderId: z.string().min(1), shopId: z.string().min(1), amount: z.number().int().nonnegative(), ts: z.number().int(), }); type OrderEvent = z.infer; const db = DynamoDBDocumentClient.from(new DynamoDBClient({})); // Idempotent write: skip if orderId already exists async function saveOrder(e: OrderEvent) { try { await db.send( new PutCommand({ TableName: process.env.TABLE_NAME!, Item: e, ConditionExpression: "attribute_not_exists(orderId)", }), ); } catch (err) { if ((err as Error).name !== "ConditionalCheckFailedException") throw err; } } // Only failed messages are returned for SQS to retry export const handler: SQSHandler = async (event): Promise => { const batchItemFailures: SQSBatchResponse["batchItemFailures"] = []; for (const record of event.Records) { try { await saveOrder(OrderEvent.parse(JSON.parse(record.body))); } catch { batchItemFailures.push({ itemIdentifier: record.messageId }); } } return { batchItemFailures }; }; ``` ## Why does this approach suit an AI? - **Infrastructure is code.** Configuration lives in files, so an AI reads, edits and diffs it through git. - **Few parts, few ways to go wrong.** Serverless scales itself. An AI has no machine counts or clusters to tune. - **SQL is a shared language.** An AI writes it well and you check the result with one query. - **Retries are safe.** Idempotent writes and a dead-letter queue allow re-runs with no duplicates. - **It runs before it deploys.** `wrangler dev` runs the Worker, Queue and D1 locally. - **Cost has a ceiling.** Paying by use keeps mistakes cheap. Still turn on cost alerts. ## How do you hand this job to an AI? Copy this, fill in the blanks and give it to the AI: ```text Build an event pipeline with these requirements. Event: [field names, types, which field is the unique key, which is event time] Volume: [events per day, at peak] Acceptable delay: [for example: under 10 seconds for the orders table, daily for the report] Platform: [Cloudflare Workers + Queues + D1, or AWS Lambda + SQS + DynamoDB] Cost ceiling: [amount per month] Required: - Write the event contract in Zod and check it at the door. - Every write is idempotent on the unique key. - A dead-letter path for messages that keep failing. - No permission to delete real data. Tokens cover only the resources needed. Prove it before reporting done: 1. Run locally with sample events. 2. Send one event twice and show the result is not doubled. 3. Send a malformed event and show it is rejected. 4. Print the result with one SQL query. ``` ## Common mistakes - **Forgetting that cron runs in UTC.** The schedule ends up hours off from what you expect. - **No dead-letter path.** Failed messages retry forever or vanish. Cap the retries. - **Giving an AI too wide a token.** Separate dev and prod, and do not grant deletion of real data. - **Running several SQL statements with `--command`.** `--file=schema.sql` is safer. - **Not measuring how far behind you are.** Watch dead-letter counts and delay, so an AI can spot problems from them. ## Check what you learned 1. You are starting out with about 100,000 events a day, and an AI will help run things. What is the sensible start? Show the answer: A Worker or Lambda, a queue and a scheduled job. At this scale, serverless with a queue and cron is enough, cheap, gives an AI few things to get wrong and is easy to try. Move to Kafka and Flink when measured numbers call for it. 2. Why does the write in the sample use ON CONFLICT DO NOTHING? Show the answer: So handling a message again does not duplicate data. A queue can deliver a message more than once. An idempotent write makes retries always safe. 3. Which time zone do Cloudflare Cron Triggers use? Show the answer: UTC. Cron runs in UTC, so 02:00 in Vietnam is 19:00 UTC the day before. ## Sources - [Cloudflare Queues: pricing](https://developers.cloudflare.com/queues/platform/pricing/) - [Cloudflare Workers: pricing](https://developers.cloudflare.com/workers/platform/pricing/) - [Cloudflare Pipelines: pricing](https://developers.cloudflare.com/pipelines/platform/pricing/) - [AWS Lambda: pricing](https://aws.amazon.com/lambda/pricing/) - [Amazon SQS: pricing](https://aws.amazon.com/sqs/pricing/) - [Amazon Kinesis Data Streams: pricing](https://aws.amazon.com/kinesis/data-streams/pricing/)