Nvidia has published a set of eight design guidelines for sparse attention, arguing that the real long-context bottleneck has moved from the attention math itself to the small component that picks which words to look at. Timmy Liu wrote the post on the Nvidia Technical Blog, dated November 17, 2026.
Some background first. Attention is the step where a language model compares each new token (a word or word fragment) against everything that came before. At 128K tokens of context, that comparison can eat most of a layer’s prefill time, the stage where the model reads your prompt before it starts answering. Sparse attention cuts the work by letting each query look at only a few thousand of the most relevant earlier tokens.
The catch is that something must find those tokens. That job belongs to the indexer, a small attention-like scorer that still rates every key in the context. Nvidia’s measurement of DeepSeek V3.2 shows the scale of the problem: the indexer and the Top-k step (which keeps only the highest-scoring keys) take 47% of a prefill layer at 128K tokens. The cost did not vanish. It moved.
Nvidia swept the indexer’s settings using a production FP8 kernel, a version of the code that stores numbers in eight bits to save memory and time. Several results are worth knowing if you build or buy inference.
Speculative decoding, where a model drafts several tokens and then checks them in one pass, turns out to be nearly free for the indexer. Decoding is limited by how fast the key cache can be read from memory, so extra query rows add arithmetic but no new reading. One pass can handle up to roughly 200 rows, which is six query tokens at a group size of 32 and 26 at a group size of 8. Over that range, runtime rose only 1.37x and 1.87x while the work grew 6x and 26x. Past about 200 rows, runtime steps up.
Query heads behave differently in the two phases. Decode runtime moved only 2% while the group size grew 16x. Prefill, which is limited by raw compute, grows with the group size once it passes about 16.
Cross-head reduction matters too. This means folding the scores from many query heads into a single score per key. With more than one key head, decode slowed by 7.6x going from one to eight, because each extra head forces another full read of the cache. Nvidia recommends one key head with cross-head reduction. Two designs it lists, MiniMax’s MSA and MoBA, skip that reduction and give up the benefit.
The other lever is scoring fewer keys. Halving the compression ratio doubles indexer work in both phases, so compressing the indexer’s keys, indexing hierarchically, or sharing one index across layers are the only ways to shrink it. Top-k follows the same logic: its cost tracks how many keys are scored, not how many are kept. Nvidia’s Guess-Verify-Refine Top-k in TensorRT LLM exploits the fact that picks barely change between decode steps. It treats the last step’s picks as a guess and checks them in a single pass. On DeepSeek V3.2 and DeepSeek V4 decoding, that beats a standard radix select by 2.1x to 2.4x.
Block-level selection is the last piece. Picking contiguous blocks of keys lets the core attention kernel fetch them in bulk and share each fetch across queries. Scattered single tokens cannot. At a fixed budget of 2,048 selected keys, halving the block size nearly doubled runtime. Nvidia’s guidance is to prefer large blocks and a large group size.
One caution is built into the post. It is a hardware vendor’s primer, and its measurements come from Nvidia’s own kernels on Nvidia’s own GPUs, so the rules describe what Nvidia silicon rewards. Models tuned this way will run best on that silicon. That is useful to know and also convenient for the seller.
The practical consequence is on the model side. If your team is training or fine-tuning a sparse model for long prompts, the eight rules give you a checklist to run before you lock the indexer design, and a 47% prefill share is a number to beat when you benchmark.
Based on “Co-Designing Sparse Attention for Efficient Long-Context Inference” by Timmy Liu, Nvidia Technical Blog (developer.nvidia.com), published November 17, 2026.