Running the same AI model well on an Nvidia chip, an AMD chip, and a smaller accelerator from a company like IBM has usually meant writing the code three separate times. Engineers behind vLLM, the widely used open source engine that serves large language models in production, say they have built a way around that problem: a set of “hardware agnostic” layers designed to work at close to full speed no matter which chip is underneath.
The reason this is hard comes down to how chips actually execute math. A GPU or accelerator gets its speed from custom, low level code called kernels, written to exploit that specific chip’s memory layout and circuitry. Code tuned for an Nvidia GPU generally will not run efficiently, or sometimes at all, on a chip from AMD, Google, or IBM without a separate rewrite. As AI labs ship models with increasingly custom internal designs, from Mixture of Experts routing to novel attention mechanisms for handling long documents, each new architecture adds another version that needs to be hand tuned per chip. The vLLM team’s blog post, published on the PyTorch project’s site, says newer frontier models have started bypassing vLLM’s shared abstractions entirely in favor of these hand tuned, chip specific paths, since that is the only way to hit top speed on the latest Nvidia Blackwell hardware.
That approach works for teams racing to squeeze out every millisecond on brand new Nvidia clusters. It is a problem for everyone else: makers of alternative chips, owners of older GPUs still in production, and any team maintaining a model outside vLLM’s own codebase, since they would otherwise be stuck rebuilding those hand tuned versions themselves for each accelerator.
The new hardware agnostic layers, described by the authors as still in progress, are built to satisfy four conditions at once. They must compile through PyTorch’s full graph compiler rather than needing custom fast paths, they must let outside hardware makers plug in overrides for their own chip, they must stay isolated from the chip specific code so the two efforts do not collide, and they must be written in portable formats, mainly plain PyTorch and the Triton and Helion programming languages, so the same code can target multiple accelerator families without a rewrite.
To check whether a portable version could keep pace with hand tuned code, the authors ran models including Gemma 4, Qwen3, and Granite 4.2 on Nvidia H100 GPUs, comparing the new layers against vLLM’s existing, chip specific implementation. According to the post, the hardware agnostic version reached total token throughput equal to 96.6 percent of the native implementation, measured as a geometric mean across the three models. That is the authors’ own benchmark, run on their own hardware choice, and it has not been independently verified. The authors also credit the IBM Spyre accelerator project for validating the approach in production, giving the result a second source beyond the blog’s own claims, though Spyre remains a narrower deployment than mainstream Nvidia and AMD GPUs.
The bigger signal is what this says about where AI infrastructure competition is actually happening. Model providers keep publicizing new architectures and benchmark scores, but the software layer that decides whether a chip maker outside Nvidia can realistically run those models is just as consequential, and far less visible. A hardware agnostic path that holds its performance closes one of the main reasons buyers default to Nvidia: the fear that anything else means rewriting your inference stack from scratch.
Teams evaluating non Nvidia inference hardware, including IBM Spyre, AMD accelerators, or Google TPUs, should track vLLM’s public RFC on this effort before locking in a chip choice for next year’s deployment, since the feature is not yet the project’s default path.
Reported by Thomas Parnell and Thomas Ortner of IBM, Richard Zou of Meta, and Harry Mellor of Hugging Face on the PyTorch blog.
