Why AMD Beat Nvidia on Cost Serving a 2.8-Trillion-Parameter Model

Wafer, an AI inference startup, reported serving Moonshot AI’s 2.8 trillion parameter Kimi K3 model on AMD’s MI355X GPUs at 952 tokens per second per node, a result the company says beats its own Nvidia Blackwell deployments on cost per token. The claim matters because AMD’s chips have mostly proven competitive on smaller open-weight models. Kimi K3 is the first case Wafer has published where a specific hardware trait, not just software tuning, appears to decide who can serve the model economically at all.

That trait is memory capacity, and the mechanism is straightforward once the sizes are laid out. Kimi K3’s weights alone need more than 1.5 terabytes of VRAM before any cache is set aside for a one-million-token context window. AMD’s MI355X carries 288 gigabytes of memory per GPU, so a single eight-GPU node holds 2.3 terabytes, enough to fit the model with room for the cache. Nvidia’s B200, at 192 gigabytes per GPU, tops out at 1.5 terabytes per node, just short. Wafer had to split the B200 deployment across two nodes and run tensor parallelism across 16 GPUs, which forces every decoding step to synchronize weights over a network link rather than the much faster interconnect inside a single box. Fitting a model in one node avoids that network tax entirely, which is the actual reason memory capacity translates into throughput.

The numbers reflect that split. On a benchmark using 1,024 input tokens and 400 output tokens, eight MI355X GPUs in one node produced 952 tokens per second in aggregate and 118 tokens per second on a single request. The two-node B200 setup produced 498 tokens per second in aggregate and 90 tokens per second single-stream, despite using twice as many GPUs. Nvidia’s newer B300, which also carries 288 gigabytes per GPU and therefore also fits Kimi K3 on one node, still won outright on raw speed: 1,568 tokens per second aggregate and 172 single-stream.

Performance per dollar flips that ranking. Wafer priced the three chips at $2.50 an hour for the MI355X, $4.25 for the B200, and $6.00 for the B300, citing rates from the GPU marketplace gpus.io rather than list prices from AMD or Nvidia. At those rates, the MI355X returned 48 tokens per second per dollar of GPU time, against 33 for the B300 and 7 for the B200. Marketplace spot pricing moves independently of enterprise contract pricing, and a buyer negotiating directly with Nvidia or AMD could see a materially different ratio. The batch configuration and software stack are specified in enough detail to reproduce; the dollar figures are not the accelerator makers’ own numbers.

Wafer is also candid, to a point, about how much engineering the MI355X result required. Kimi K3 ships without the small “draft” model that speeds up generation through speculative decoding, so Wafer used an external draft model instead. On AMD’s ROCm software stack, that path initially crashed because a kernel function the code called was never defined for AMD’s chip architecture; Wafer wrote a short replacement and recovered most of the missing speed. A separate bug slowed the time before a model produces its first token: an attention kernel refused to run because Kimi K3’s configuration produced 12 attention heads per GPU, and the kernel only accepted counts of 4, 8, or multiples of 16. Padding the head count to 16 fixed a roughly 2 to 3 times slowdown in that stage. Both problems required Wafer’s own patches, which cuts against any read of the results as proof AMD’s software now matches Nvidia’s out of the box.

Wafer sells inference capacity and has now published back-to-back posts showing strong AMD results, having run a similar comparison on the GLM5.2 model in July. A vendor demonstrating that the hardware it operates competes with the market leader has a direct commercial reason to publish favorable framing of that comparison, even where its underlying methodology is transparent.

AI Insiders has covered Kimi K3’s open weights and the inference stack building around it this past week.

For teams evaluating Kimi K3 or models of similar size, the deciding question is not raw GPU speed but whether the model’s weights fit inside a single node’s memory: that single fact determines whether cross-node network overhead enters the equation at all, and it is worth checking against a vendor’s own priced-out quote rather than a marketplace rate.

Wafer detailed the benchmark and the engineering behind it in a company blog post published July 31, 2026.