Inferact, an AI infrastructure startup, released open source software on September 23 that it says lets Google’s TPU v7 chips generate AI responses faster than Nvidia’s GB200 chips do. The claim rests entirely on Inferact’s own benchmarks, run on a large open source model called Kimi K3.
The stakes are straightforward. Nvidia’s chips are the default choice for running AI models at scale, and that default carries pricing power. If a software layer can make a rival chip competitive on speed, cloud buyers gain leverage, and Google Cloud gains a stronger pitch for AI workloads it does not already own by default.
The software, called tpu-megakernels, targets what engineers refer to as “decode,” the step where a model generates each word of its answer one at a time. Normally, that step runs as a long relay of small programs called kernels: one to fetch data, another to compute, another to store the result, repeated hundreds of times per response. Each handoff between kernels can leave the chip’s memory system briefly idle. Inferact’s fix combines the entire relay into a single continuous program, a “megakernel,” so the chip keeps pulling the next chunk of model data into fast on-chip memory while it is still finishing the current one.
That approach depends on how much fast on-chip memory a chip has. TPU v7 gives each processing core 64 mebibytes of what Google calls VMEM, according to Google’s own published specs. Nvidia’s GB200 splits a much smaller pool of on-chip memory, about 38 mebibytes total, across 152 separate processing units, leaving little room to stage data ahead of time. Inferact says that difference in memory layout, more than the two chips’ raw compute numbers (which are close on paper, per Google’s and Nvidia’s own spec sheets), is what let its software pull ahead.
On the numbers Inferact reports, the gap is largest for a single user waiting on one response: 249 tokens per second on 16 TPU v7 chips versus 127 on 16 GB200 chips, roughly double. Using a speed-up technique called speculative decoding, where a smaller draft model guesses ahead and the main model checks the guesses, Inferact says its setup reached over 700 tokens per second per user on the TPUs against 452 on the GB200 baseline. The advantage narrows as more requests are handled at once, from about double at the smallest batch size to roughly 1.36 times faster at a batch of eight.
None of these figures come from an independent lab. Inferact measured its own software against a GB200 configuration that vLLM, the popular open source inference engine, publishes as its recommended recipe, not against whatever the best available tuning on Nvidia hardware might achieve. The company says it validated output quality on two public benchmarks, GPQA-Diamond and GSM8K, but has not published results from a third party running the same comparison independently.
Inferact frames the release as a first step rather than a finished product. Its current design is tuned to one specific arrangement of 16 chips and would need new work to support other cluster layouts, and the company says higher-traffic, multi-user workloads expose different bottlenecks than the small-batch decoding it tested. The code and benchmark scripts are open source, which means outside teams can reproduce or challenge the numbers directly rather than take Inferact’s word for it.
For any team running large open-weight models like Kimi K3 on rented GB200 clusters, tpu-megakernels is now something to test against a real workload before the next infrastructure contract is signed, rather than a result to take on faith from a vendor’s own blog post.
Reported by Inferact in its own engineering blog, published September 23, 2026.
