What are the ways to stop a model rereading everything?

Someone reads a whole library and is then asked one question. There are three ways to make that less work: keep only a thin notebook, keep the whole library and check the catalogue every time, or keep the whole library and search it floor by floor. AI models go in those same three directions. This lesson puts them side by side on the same context.

Try it: one question, five ways to answer it#

The top strip shows where the true attention sits. Each strip below shows the part of the context one method reads. The table says how many comparisons it costs and how much attention it keeps, compared with reading everything.

Try this order:

1. Keep the defaults: 16K tokens, with what you need 60% back. The window keeps 0% of the attention. Scoring each key keeps 97%, but costs nearly 17 thousand comparisons, more than reading everything.
2. Drag the slider down to 1%. What you need is right here, and the window keeps about 91% for only 512 comparisons. A window is not bad, it is just blind to anything far away.
3. Go back to 60% and switch to 64K. The two rows that score things grow about 3 to 4 times. The tree row grows less than 2 times, from 636 to 1,096.
4. Look at what the tree keeps: about 85% at 16K, and only about 62% at 64K with the same setup. That is its price, and lesson 4 comes back to it.

The demo is a simplified model with made-up keys. A real indexer's score is cheaper than a full attention step, but the number of scores still grows with the length. The shape is what to look at.

Way 1: compress everything into a fixed-size state#

This is the cheapest and the most forgetful. A sliding window only keeps the last N tokens. State-space models such as Mamba keep a fixed-size state and update it along the sequence. This way, reading is linear and each written token costs the same.

The limit is capacity. Useful information keeps growing with the text, and the space does not. It is like reading more and more books with exactly 10 sheets of notes: enough at first, and later something new has to overwrite something old. A 2024 experiment showed transformers beating state-space models at copying a sequence, exactly the kind of task that needs precise memory.

So "hybrid" models mix the two: most layers use a fixed state and a few use full attention. According to the ZHA paper, at long enough context those few attention layers still dominate the cost.

Way 2: keep the whole context, score each key, read only the best#

This is far more accurate, but it only moves the cost elsewhere. Sparse attention keeps the whole KV cache and lets each query pick what it reads. The strength is that one query reads here and another reads there.

The catch is: how do you know which 1% is worth reading? The indexer has to score every key. The final attention is cheap, but the indexer grows with the length just as the old attention did. It can be made lighter with fewer attention heads or lower precision, but the ZHA paper cites figures showing that near 1 million tokens, 60 to 90% of the time in these layers is still spent on indexing.

Way 3: score blocks instead of keys#

Summarizing each group of keys into a block and scoring the blocks is cheaper than scoring keys, but every block still gets scored. MiniMax Sparse Attention (MSA) is one example: the ZHA paper describes the configuration it compares against as choosing 16 blocks of 128 tokens each.

The number of blocks still grows with the length. The code below compares the two ways, both reading 512 keys at the last step:

// indexers.ts: node indexers.ts
const everyKey = (n: number) => n + 512; // score every key, then read 512
const everyChunk = (n: number) => (n / 64) * 4 + 512; // score every chunk with 4 summaries, then read 512

for (const n of [1e4, 1e6, 1e8]) {
  const a = everyKey(n);
  const b = everyChunk(n);
  console.log(`${n} tokens: ${a} vs ${b}, ${(a / b).toFixed(1)}x cheaper`);
}
console.log(`100x the tokens costs ${(everyChunk(1e8) / everyChunk(1e6)).toFixed(0)}x more with chunks`);

On 3 November 2026, on Node 26.11.1, the result was:

10000 tokens: 10512 vs 1137, 9.2x cheaper
1000000 tokens: 1000512 vs 63012, 15.9x cheaper
100000000 tokens: 100000512 vs 6250512, 16.0x cheaper
100x the tokens costs 99x more with chunks

Blocks are up to 16 times cheaper, and the gain stops at 16. A context 100 times longer still costs 99 times more. That is what a constant-factor speedup means: the cost line moves down, but its slope does not change.

Way 4: search through several levels#

The slope only changes if the search itself is sparse. Summarize blocks into larger blocks, summarize those, then walk from the top and open only the branches that look right. When a branch is skipped, no key beneath it needs scoring. The ZHA paper names two earlier multi-level methods, PISA and HiP, which reach log-linear complexity but, according to the paper, have not been shown to work on hard reasoning and coding tasks. The next lesson goes into how such a tree works.

Which way fits which job?#

Common mistakes when thinking about this#

• Reading "reads only 1%" as "costs only 1%". Add the cost of finding that 1%. This is the most common mistake when reading sparse attention announcements.
• Deleting what gets little attention. The ZHA paper measures that the union of what many questions read covers nearly the whole cache in time. Nobody reading it today does not mean nobody needs it tomorrow.
• Treating RAG and sparse attention as the same thing. RAG picks passages once, outside the model. Sparse attention picks per query in each layer, while the whole context stays in the cache.
• Comparing two methods at one length. A method 16 times cheaper can win at 100 thousand tokens and lose at 100 million. Ask about the slope, not just one point.

From these limits the ZHA paper draws two requirements for anything that truly escapes quadratic cost: each query must choose what it reads, and the part being read must be allowed to grow, but more slowly than the text. The next lesson looks at a design that tries to meet both.