# Should you trust a 400× speedup? Reading ZHA's results properly

> Separate what was measured from what was projected in the ZHA paper, see why deep trees lose attention, and take a checklist to the next paper.

Canonical URL: https://learn.chabulab.com/ai/reading-zha-results-honestly. Facts checked: 2026-11-03.

- ZHA's accuracy was measured up to 256K tokens and came close to full attention. The hundreds-of-times speedups at 1 billion tokens come from a calculation, and no optimized GPU kernels have been released.
- The configuration behind the speed projection is lighter than the one in the headline accuracy table, and it scores 1 to 6 points below full attention on most tests.
- With a fixed threshold p, deeper trees lose attention, and raising p with depth costs more. The paper calls this a narrow path, and it has not been verified at very long context.

"400 times faster" is the most shareable number in the ZHA announcement, and the one to read most carefully. Not because it is wrong, but because it is a completely different kind of evidence from the scores next to it. This lesson separates those kinds, then gives you a list of questions to use on the next paper.

## Try it: what was measured, and what was only projected?

The chart below puts every published speed number on one line and shades the range with no accuracy measurement. Pick each length to read the evidence there.

*Each dot is a published point. The area right of the dashed line has no accuracy measurement. Pick a length to read the details.*

What to notice: every truly impressive point sits in the unmeasured range.

## What was actually measured?

Accuracy was really measured, on Qwen3.6-35B-A3B, with two configurations at p = 0.95: a deep tree (chunks of 4, 6 levels) and a wide tree (chunks of 256, 1 level). The numbers below come from Figure 5 of the [paper](https://arxiv.org/abs/2610.42112).

| Test | Full | Deep ZHA | Wide ZHA | MSA |
| --- | --- | --- | --- | --- |
| AIME 2026 | 93.1 | 91.2 | 90.8 | 89.6 |
| SWE-bench Verified | 67.4 | 67.4 | 68.4 | 65.6 |
| Terminal-Bench 2.1 | 49.9 | 46.7 | 53.3 | 45.4 |
| MMLU-Pro | 84.9 | 85.3 | 85.2 | 85.4 |
| GPQA Diamond | 84.8 | 85.0 | 85.0 | 83.3 |
| RULER-13 (256K) | 96.0 | 93.3 | 96.2 | 77.1 |
| LongBench v2 | 58.0 | 63.4 | 62.4 | 50.7 |

The MSA column stands out on RULER-13 and LongBench v2: reading about 1% of keys on LongBench v2, deep ZHA scores 12.7 points higher. The paper also says it trained MSA's indexer only through the KL warm-up stage, with 40 billion tokens, so this is not the fullest possible MSA.

On the Nemotron model, which has no positional encoding, RULER-13 at p = 0.95 drops 25.3 points with the deep tree and 10.2 with the wide tree. How the method does depends on the base model.

## Is the fast configuration the one in that table?

No, and this is the detail easiest to miss. The speed projection uses a much lighter configuration: one reference query, chunks of 8, p = 0.75 (R1 C8). Its scores at 2 levels are in Figure 15 of the paper:

| Test | Full | R1 C8 L2, p = 0.75 | Difference |
| --- | --- | --- | --- |
| AIME 2026 | 93.1 | 89.4 | −3.7 |
| SWE-bench Verified | 67.4 | 66.0 | −1.4 |
| Terminal-Bench 2.1 | 49.9 | 44.0 | −5.9 |
| MMLU-Pro | 84.9 | 81.8 | −3.1 |
| GPQA Diamond | 84.8 | 81.4 | −3.4 |
| RULER-13 (256K) | 96.0 | 94.5 | −1.5 |
| LongBench v2 | 58.0 | 60.5 | +2.5 |

Still quite good, but below full attention on six of the seven tests. And those numbers are at **2 levels**. The projection for 1 billion tokens uses **5 levels**, a depth where this light configuration has no quality measurement. Also, the growth exponents are carried over from a C4 R16 configuration rather than measured directly at that length.

The paper does not hide any of this. It says so plainly. A headline just has no room for it.

## Why do deeper trees lose attention?

Because the threshold p multiplies across levels. Keeping 90% at each level keeps about 0.9⁶ ≈ 53% after six levels. In the last lesson you saw the "keeps" column fall. This is why, and you can measure it:

*Each line is one way to set the threshold. Drag p to see how the part kept and the cost change as the context grows.*

On this lesson's small model, with p = 0.9, averaged over 30 questions, measured on 3 November 2026:

| Length | Levels | Fixed p: kept / compared | Rising p: kept / compared |
| --- | --- | --- | --- |
| 1K | 2 | 82% / 685 | 90% / 892 |
| 4K | 3 | 75% / 603 | 90% / 1,039 |
| 16K | 4 | 68% / 657 | 88% / 1,800 |
| 64K | 5 | 56% / 811 | 82% / 3,688 |
| 256K | 6 | 53% / 794 | 79% / 3,649 |

The table shows both sides of the trade-off. With a fixed p, the cost barely moves, but the part kept shrinks towards a fixed size, and the tree catches the same disease as fixed memory. Raising p with depth keeps more, but costs several times as much. Raise it too far and the upper levels open nearly everything, and the cost heads back to quadratic.

The authors call this a narrow path between two failures, and say the way forward depends on the data: the part read should grow with the predictive information actually in the text. Their data on text stops at 256K tokens, and they write plainly that extrapolating to hundreds of millions or billions of tokens could fail.

## Is ZHA really better than full attention?

On LongBench v2 it is, 63.4 against 58.0, and this is the most interesting part. Full attention has to spread probability over a great many distracting tokens, so at long context it can get diluted. The authors raise the possibility that sparsity also filters noise, and that being sparse may be necessary rather than just thrifty.

That is a hypothesis, because on RULER-13 the deep tree is below full attention. One test on one model does not prove a law.

## How should you read an announcement like this?

Use the questions below on the next paper, even if it has nothing to do with attention.

| Question | Why ask | For ZHA |
| --- | --- | --- |
| Was this number measured or projected? | A calculation can be right, but it is not yet evidence | Speed: a roofline model. Accuracy: measured |
| Measured up to where? | Extrapolation can break outside the measured range | Accuracy to 256K, projected speed to 1 billion |
| Do speed and quality use the same configuration? | It is easy to pair the best numbers of two different setups | No: R1 C8 for speed, R16 for the headline table |
| Is there code to run it again? | Without it, nobody can check independently | No optimized kernels released |
| How was the competitor set up? | Was it given a fair run? | MSA only went through its warm-up stage |
| Where does it fail? | A good paper says so itself | Nemotron, small models, very long context |

Our conclusion: ZHA is a well-founded idea with notable measured results up to 256K tokens. It is not yet evidence that a model will read a billion tokens well. Wait for someone to run it for real on a GPU, at millions of tokens, with quality scores next to the timings.

## How do you hand this to an AI?

Copy the text below, paste in the paper or announcement, and give it to an AI:

```text
Here is a paper or announcement about a new AI method: [paste the content or a link]

Split its claims into a table with these columns:
- Claim (the exact numbers)
- Kind of evidence: measured, calculated from a model, extrapolated, or only a hypothesis
- Conditions: model, configuration, length, hardware
- Where in the document (section or figure)

Then answer:
1. Was the most prominent number measured directly?
2. Were speed and quality measured on the same configuration? If not, how do they differ?
3. Where does the measured range stop, and what lies outside it?
4. Was the competitor trained and configured fully?
5. What limits do the authors state themselves?
6. What experiment would test the biggest claim?

Use only the document I gave you. If it does not say, write "not stated" rather than guessing.
```

That ends the series. You can go back to the [first lesson](/ai/what-is-attention-and-why-it-costs) to see again why all of this starts from a triangle.

## Check what you learned

1. The paper projects ZHA at 373 to 417 times faster at 1 billion tokens. Where does that number come from?
   Show the answer: A performance model, calibrated with measurements at shorter lengths. The paper uses a roofline model and says plainly that it does not release optimized kernels. The accuracy measurements stop at 256K tokens.
2. The threshold p is 0.9 at each level and the tree has 6 levels. If each level were independent, about how much attention is left?
   Show the answer: 53%. The levels multiply: 0.9 to the power of 6 is about 0.53. On this lesson's small model, the measured value at 6 levels is also about 53%.
3. Why not just pick a very high p, such as 0.999, to be safe?
   Show the answer: A very high p opens nearly everything at the upper levels, so the tree stops helping and the cost heads back to quadratic. A tree only saves work when it can skip many branches. Opening nearly everything at the top leaves nothing to skip.
4. Deep ZHA scores 63.4% on LongBench v2, above the 58.0% of full attention. How should you read that?
   Show the answer: Filtering may reduce noise, but this is still a hypothesis, from one test on one model. The authors raise the possibility that sparsity removes noise, and they say may. On another test, RULER-13, deep ZHA is below full attention. One result is not enough to generalize.

## Sources

- [Lufkin et al. (2026): Subquadratic Attention with a Hierarchical Tree-Structured Indexer](https://arxiv.org/abs/2610.42112)
- [Zyphra: ZHA, Zyphra Hierarchical Attention](https://zyphra.com/our-work/zyphra-hierarchical-attention)
- [Lai et al. (2026): MiniMax Sparse Attention](https://arxiv.org/abs/2606.13392)