Every inference server has to guess, before a request finishes, how many tokens it will eventually generate. Get the guess wrong and the server either reserves far more GPU memory than it uses or runs out mid-generation. According to engineer Nick Gustafson, writing on his blog The Gustafson, production serving systems that reserve a fixed worst-case block per request typically end up using only 20 to 40 percent of the KV cache memory they hold onto, a figure he attributes to Kwon et al.’s original vLLM paper from SOSP 2023.
What actually fills that reserved space is a per-request cache: each attention key and value the model computed for a prompt, kept around so the next token does not force a full recalculation. Push the context window out far enough and this cache overtakes the model’s own weights as the biggest thing sitting in GPU memory, simply because each additional token adds another slice to store. That ordering matters for anyone budgeting a serving cluster: the waste is not happening in a secondary resource, it is happening in the single largest line item on the GPU.
PagedAttention, the technique behind vLLM and now standard in engines like Hugging Face’s TGI and TensorRT-LLM, fixes this by applying a decades-old operating-system idea to that cache. Instead of reserving one contiguous block per request sized for the worst case, the GPU holds a shared pool of small fixed-size physical blocks, 16 tokens each in the common configuration. Each request keeps a small block table mapping its logical sequence position to wherever its data actually sits in the pool, the same relationship a page table has to physical memory frames. A request that only ever generates 47 tokens takes three 16-token blocks, not the 2,048-slot allocation a naive system would set aside.
The mechanism only works because attention kernels already process keys and values in chunks rather than as one dense contiguous span. FlashAttention, for instance, tiles K and V into blocks sized to fit GPU SRAM and walks them iteratively. PagedAttention lines its block size up with that existing tiling, so the kernel just adds one lookup, from logical block to physical block, before fetching each chunk. Gustafson describes the added overhead as landing in the single digit percent range, with 16 tokens per block emerging as the setting vLLM, TGI, and TensorRT-LLM converged on independently.
The technique also supports copy-on-write, the same trick operating systems use when a process forks. Multiple requests that share an identical prefix, a system prompt prepended to every user turn, for example, can have their per-request tables reference identical physical blocks rather than each duplicating the data. A block is only cloned the moment one of the sharing requests needs to write into it. Gustafson estimates that a 2,000-token system prompt shared across a hundred concurrent users saves roughly 200,000 tokens of cache versus copying it per request, tens of gigabytes on a 70B model in fp16.
None of this changes model output. Attention produces identical logits whether the K and V tensors sit in one contiguous span or are scattered across a block pool; only the memory-fetch path changes. The measured effect Gustafson cites, drawn from the vLLM paper, is cache utilization rising to roughly 96 percent and throughput improving two to four times per GPU compared with the naive contiguous approach.
For any team serving models at long context, this is a budgeting question before it is a technical one: an inference stack that still allocates KV cache contiguously is paying for GPU memory it never uses, and switching to a paged engine is closer to a cost-reduction decision than a performance tweak.
Nick Gustafson explained the mechanism in a post on his blog, The Gustafson (thegustafson.com); the piece carried no publication date at the time of this rewrite.
