What is attention, and why does long text cost AI models so much?

You reach the word "it" in a story and your eyes jump back to check what it refers to. That is quick, because you only look at a few places. A language model does something similar with attention, except that in its basic form it looks back at everything, every time. This lesson shows you both the mechanism and that price.

Try it: which words does a word look back at?#

Click a word to see how it splits its attention over the words before it.

Try this order:

1. Pick was hungry. About 78% of the attention lands on "The cat", because the hungry one is the one chasing.
2. Change the ending to …because it was scared. About 85% moves to "the mouse". None of the earlier words changed, only the question did.
3. Pick it. The attention splits 45% and 45% between the two animals, because nothing separates them yet.

The demo uses vectors set by hand so the mechanism is easy to see. A real model learns them from data.

What steps does attention take?#

Attention matches what a token is looking for against the label of every earlier token, turns the scores into shares, then blends the content by those shares. Think of a librarian: you say what you need, the librarian checks the label on every book, then hands you the contents of the closest matches.

The match is a dot product of two vectors. In this series we call it one comparison, and it is the unit of cost throughout. The original transformer paper calls this method scaled dot-product attention.

Why does twice the length mean four times the work?#

Because each token is compared with every token before it. Token 8 makes 7 comparisons, and token 1,000 makes 999. Added up, n tokens take n(n-1)/2 comparisons, so the cost grows with the square of the length. That is quadratic cost.

Each square below is one comparison. Press Double it and watch the triangle grow.

A triangle of 64 tokens can still be drawn. The table below is the part that cannot. The time column assumes a machine doing 1 billion comparisons per second, for one attention layer and one attention head. That number is an assumption to make the scale easier to picture, not a speed measured on any particular hardware.

How do reading and writing differ?#

The two stages cost in different ways. Prefill is when the model reads the text it was given: the total cost is quadratic. Decode is when the model writes the next token.

To avoid recomputing, the model stores the key and value of every earlier token in a KV cache. But each new token still reads that whole cache. So the paper on ZHA describes the cost of decode as memory bandwidth and storage that increase linearly with the sequence length, for every new token.

In other words, token one million costs more than token one thousand in more than arithmetic. It also has to pull a million key and value pairs through memory.

Checked by running it#

On 3 November 2026 we ran the code below on Node 26.11.1 and an Apple M4 Pro, with no outside libraries. It is a plain CPU loop, only there to show the shape of the growth.

// attention.ts: node attention.ts
const DIM = 32;

function randomKeys(n: number): Float32Array {
  const keys = new Float32Array(n * DIM);
  for (let i = 0; i < keys.length; i++) keys[i] = Math.random() - 0.5;
  return keys;
}

// Each token compares itself with every token before it.
function readAll(keys: Float32Array, n: number): number {
  let comparisons = 0;
  let sink = 0;
  for (let token = 1; token < n; token++) {
    for (let earlier = 0; earlier < token; earlier++) {
      let score = 0;
      for (let d = 0; d < DIM; d++) score += keys[token * DIM + d]! * keys[earlier * DIM + d]!;
      sink += score;
      comparisons++;
    }
  }
  if (Number.isNaN(sink)) throw new Error("unreachable");
  return comparisons;
}

let previous = 0;
for (const n of [2000, 4000, 8000, 16000]) {
  const keys = randomKeys(n);
  const start = performance.now();
  const comparisons = readAll(keys, n);
  const ms = performance.now() - start;
  const growth = previous ? `${(ms / previous).toFixed(1)}x` : "";
  console.log(`${n} tokens: ${comparisons} comparisons, ${ms.toFixed(0)} ms ${growth}`);
  previous = ms;
}

The result, where each line doubles the tokens:

2000 tokens: 1999000 comparisons, 53 ms
4000 tokens: 7998000 comparisons, 189 ms 3.6x
8000 tokens: 31996000 comparisons, 746 ms 4.0x
16000 tokens: 127992000 comparisons, 3000 ms 4.0x

Your machine will show different milliseconds, but the ratio stays near 4.

What does this change about how you use AI?#

Full attention costs the provider more the longer the text, so long context usually comes with limits or a higher price. A few habits follow from the mechanism.

Common misunderstandings#

• "Attention means the model only reads the important part." Full attention does the opposite: it scores everything, and only then gives the important part a large weight.
• "The KV cache removes the cost." The cache avoids recomputing keys and values, but a new token still reads the whole cache.
• "A machine twice as fast reads twice as far." A machine twice as fast only buys about 41% more length, because the cost grows with the square.
• "One comparison in a real model is like the one in the demo." A real model has many layers and many attention heads, so its numbers are larger. The shape of the growth is what to remember.

The next lesson looks at the ways people have tried to escape this cost, and why most of them only move it somewhere else.