Google's TPUv7 beats Nvidia on cost per token, analyst finds

Google’s newest TPU, codenamed Ironwood, delivers up to 50 percent better performance per dollar than Nvidia’s B200 and B300 chips in aggregated inference serving, according to independent testing SemiAnalysis published this week through InferenceX, its hardware benchmarking arm. The results are the first outside verification of how Google’s chip performs against Nvidia rather than only against Google’s prior TPU generations.

The comparison matters because Ironwood is the first TPU generation Google is genuinely selling to outside customers for inference rather than keeping mostly for itself. Companies can now buy the chips outright or rent them through Google Cloud, a real change after a decade in which TPUs ran almost entirely Google’s own Search, Ads, YouTube and Gemini traffic. Anthropic, the largest external TPU customer, has committed to more than one million of the chips, roughly 400,000 purchased directly and 600,000 rented through Google Cloud, per SemiAnalysis’ Accelerator Model.

The headline figures are specific and worth stating plainly. At a serving target of 100 tokens per second per user, SemiAnalysis measured Ironwood’s cost at $0.181 per million total tokens, against $0.222 for B200 and $0.276 for B300, gaps of roughly 19 percent and 34 percent. At a lower interactivity target of 20 tokens per second per user, Ironwood also produced about 5 percent higher raw throughput than either Nvidia chip. Paired with its lower modeled hourly cost, that translated into 50.4 percent more tokens per dollar than B200 and 96 percent more than B300 at that setting.

Those numbers rest on assumptions SemiAnalysis is upfront about, and readers should treat them as modeled estimates rather than a settled benchmark. The comparison uses SemiAnalysis’ own total-cost-of-ownership modeling for an external buyer, not published list prices from either vendor. Swap in Google’s internal chip-hour cost of $1.03 instead, and the advantage climbs as high as 130 percent over B300 at high concurrency, but that scenario also comes with a real tradeoff: Ironwood’s response latency stretches to 5.41 seconds at that setting, versus 2.40 seconds for B300. Ironwood also has no native support for Nvidia’s lower-precision FP4 format, so the favorable numbers hold only against Nvidia’s higher-precision FP8 mode. Google’s follow-on chip, TPUv8i, is the one built to compete on FP4.

Nvidia still wins part of the comparison outright. In disaggregated serving, where prefill and decoding run on separate hardware pools for efficiency, Nvidia’s GB300 setup holds roughly a 30 percent performance-per-dollar lead over Ironwood in the middle of the latency curve. Google has run disaggregated serving internally for years, but the external TPU software has not been optimized for it yet. SemiAnalysis expects that gap to close within months as Google and its partners finish the work.

That software gap is the real story underneath the pricing chart. Nvidia’s grip on inference was never only about chips: vLLM and SGLang, the two serving frameworks most AI companies actually deploy, treat Nvidia hardware as the default and add support for new open-weight models on it almost immediately. TPUs have run through a translation layer called TorchAX that converts PyTorch code into JAX before it reaches the chip, adding friction Google is now trying to remove with a new framework called TorchTPU, which lets developers treat a TPU as a native PyTorch device. SemiAnalysis expects TorchTPU to leave private beta and go open source around October.

Buyers are not really choosing silicon here. They are choosing a serving stack they have already built pipelines, tooling and staff around, and that choice changes far more slowly than a price list does. A 50 percent cost edge on paper will not move meaningful inference volume off Nvidia until Google’s software stack reaches the same day-zero model support developers already take for granted elsewhere.

Operators running large open-weight inference workloads should treat October’s TorchTPU release, not this week’s benchmarks, as the real test of whether Ironwood becomes a genuine alternative rather than a cheaper option nobody can easily deploy.

Figures and analysis from SemiAnalysis, published September 7, 2026 on the SemiAnalysis newsletter via its InferenceX blog.