vLLM, the open-source engine for serving language models, can now start moving a prompt’s working memory to the machine that writes the reply before the machine that read the prompt has finished. In a benchmark run by engineers at AWS and Mistral, the wait for the first word of an answer fell from about 1,744 milliseconds to about 540 at the busiest setting tested. The results appeared on vLLM’s own blog on 14 October.
Serving a model has two stages. Reading the prompt, called prefill, is heavy on raw computation. Writing the answer one small piece of text at a time, called decode, leans on memory. Some operators run the stages on separate machines so each can be sized on its own. The price is a handoff: the notes the model built while reading, known as the key-value cache, must reach the writing machine before the first word can appear.
vLLM’s default handoff is a “pull” design, built on NVIDIA’s NIXL transfer library. When reading ends, a router passes details to the decode machine, which reserves space, prepares the transfer, and then fetches the data. The new “push” design reverses the order. The decode machine reserves its space and tells the prefill machine where to write while the prompt is still being read. When both sides are ready, the prefill machine writes straight into the decode machine’s memory, and a background thread handles the paperwork.
That background thread explains the second improvement. The authors say pull mode posts its transfer setup inline on the decode worker, which stalls the generation of the answer itself. So pull costs more than a late start: it slows the reply while it streams.
The test used Qwen3-32B on one prefill node and one decode node, each an AWS P5en machine with eight NVIDIA H200 chips, linked by AWS’s EFA network. Prompts ran from 1,000 to 16,000 tokens, at 1 to 8 requests per second, with 128 output tokens each. Prefix caching was switched off, so every request had to ship its full cache. The modes were nearly indistinguishable when requests were sparse and prompts brief. At 8 requests per second with 16,000-token prompts, pull took about 1,744 ms to first token and push about 540. Time per output token (TPOT) sat near 5.6 to 5.8 ms for both under light load. At 4 to 8 requests per second with long prompts, pull climbed to about 10.4 ms while push held near 7, which is the post’s “up to 30 percent.”
The post names a trade-off, and it is the part buyers should read twice. Push forces the system to choose the decode machine early, so setup can overlap with reading. Pull lets the router choose after reading ends, which leaves room to steer traffic by how busy each machine is. The authors say pull remains the safer pick when data must flow both ways or the decode choice has to wait. In return, push lets the prefill machine free its memory sooner under comparable load. The post reports no throughput or memory cost for push, and it does not measure what losing late routing costs in a lumpy real-world traffic mix.
The evidence is also narrow. It covers one model, one prefill-plus-decode pairing, and AWS hardware, and the authors caution that other workloads and setups will give different numbers. The authors built the feature first on AWS Trainium clusters and contributed it to vLLM as accelerator-agnostic, but the numbers shown here come from GPUs. Turning off prefix caching is a fair way to isolate the transfer, though it is also the hardest case for the handoff. Production traffic that reuses cached prompts would skip some transfers, and the post does not say how much of the gain survives that.
Every latency win of this kind is a cost story. A server that holds its response times at 8 requests per second while its neighbor slips can take on more traffic before an operator orders another node. The post gives no capacity figures, so that is our inference, but it is the same arithmetic behind the compute-economics stories elsewhere in this issue: more users per machine is margin.
Switching on the feature means configuring both servers with NixlPushConnector instead of NixlConnector. The reproduction guide installs vLLM 0.29.0 and says any build with the push connector will do. The post does not label it stable or experimental. The authors have also opened a pull request, number 45211, to bring the same background threading to pull mode. If it lands, it will show how much of the gain came from push itself and how much from getting setup off the main loop, and teams that need late routing may get most of the benefit without giving it up.
Based on the vLLM blog post by Sunita Nadampalli (AWS) and Nicolò Lucchesi (Mistral) of 14 October 2026.
