Learn · Series · 4 lessons
Attention and long context
Why AI models pay more and more to read long text, the ways people cut that cost, how a tree of summaries searches a context, and how to judge the claims.
- What is attention, and why does long text cost AI models so much?Click a word to see where attention looks back, then see why doubling the length of a text means four times the work.
- What are the ways to stop a model rereading everything?Compare a sliding window, indexers and a tree on the same context, and see which forgets and which only moves the cost elsewhere.
- How does a tree of summaries search a context? How ZHA worksWatch a question walk down a tree level by level, tune top-p, try random data, then run a small version in about 130 lines.
- Should you trust a 400× speedup? Reading ZHA's results properlySeparate what was measured from what was projected in the ZHA paper, see why deep trees lose attention, and take a checklist to the next paper.