Salesforce AI Research built a KV cache eviction method that never inspects attention weights, skips any statistics drawn from value vectors, and requires zero calibration data to operate. It just keeps a uniformly random slice of the generated tokens, plus a short recency window, and throws the rest away. Across five reasoning benchmarks and two model families, that random policy matched or beat four learned alternatives designed to make the same decision more intelligently.

The KV cache is the running memory a transformer keeps of every token it has already generated, and it is what makes long reasoning chains affordable at all. Left unbounded, that memory grows with every token and eventually exhausts GPU memory. Eviction is how engineers keep it bounded: periodically decide which entries to discard so the model can keep generating. Every eviction method that came before this one spent real compute figuring out which entries mattered most, on the assumption that a smarter selection rule would preserve more of what the model needs.

The paper, “Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning” (arXiv

.03430), tested that assumption directly. The researchers call their method Random Attention, implemented as random_pp in the released code: keep the prompt intact, then keep a random per-head subset of generated tokens up to a fixed budget. At matched budgets, it matched or beat SnapKV, R-KV, VaSE, and TriAttention, four selectors that use attention scores or learned statistics to pick what survives.

The comparison ran on Qwen3-4B, Qwen3-14B, Qwen3-32B, and Phi-4-reasoning, across MATH-500, GPQA-Diamond, AIME, HMMT, and LiveCodeBench. That is the paper’s full scope: five reasoning benchmarks, two model families at four parameter sizes, all on 8x H200 GPUs. The authors do not report results outside these benchmarks or these model families, and the finding should not be read as a claim about non-reasoning workloads or other architectures.

Since Random Attention never runs a scoring step at all, each eviction round pays only for compacting the tokens themselves. In separate efficiency testing, distinct from the accuracy comparison above, Salesforce AI Research found it outran every other evictor in the group, whether measured inside a Hugging Face inference harness or on a vLLM serving stack.

A random baseline matching tuned heuristics is a familiar signal in machine learning: when a simple, information-free method holds its own against a subfield’s engineered selectors, it usually means the selectors were extracting less signal than their complexity implied, not that the task is easy. Results like this tend to save more engineering time than GPU time, because the real cost of the learned selectors was the calibration data, the scoring passes, and the maintenance burden required to keep them tuned to each new model, not the milliseconds they added to inference.

This is a preprint accompanied by released code, not a peer reviewed publication, and the benchmark numbers are self reported by the paper’s authors. Teams running long reasoning chains in production on Qwen3 or Phi-4-class models have a low-cost baseline worth testing against their current eviction policy before investing further in learned selection.

Salesforce AI Research, “Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning,” arXiv

.03430, published via the SalesforceAIResearch/Random-Attention GitHub repository.