LMSYS Cuts H20 Decode Gap With Nvidia B300 to 1.42x

LMSYS, the research group behind the SGLang inference engine and the Chatbot Arena leaderboard, published a serving engineering study showing that DeepSeek-V4-Pro, a Mixture-of-Experts model with 1.6 trillion parameters, needs four distinct deployment configurations to run efficiently on Nvidia H20 GPUs, not one. Getting that choice wrong is not a rounding error: a team that copies one reference topology onto every workload either over-provisions GPU capacity for short requests or blows through its latency service target once real concurrency and long context hit the system. LMSYS treated topology selection as an empirical profiling exercise rather than a spec-sheet lookup, and the resulting numbers close much of the gap between an export-restricted chip and Nvidia’s newest silicon.

H20 is the export-compliant successor to the H100 that Nvidia sells into markets without access to frontier chips. It has no native FP4 Tensor Cores and trails Nvidia’s B300 by roughly 45.6 times in peak Tensor Core throughput on the precisions each chip uses in production, plus 1.67 times in memory bandwidth. LMSYS’s position is that the chip’s 900 GB/s NVLink and available HBM still support demanding serving targets if the software path is tuned to the hardware instead of ported from a Blackwell deployment.

The team split serving by role. Prefill, which holds no long-lived state, ran on H20-96GB nodes and switched topology by context length: PP2-CP8-TP8 for shorter prompts, PP4-CP8-TP8 once inputs passed roughly 128,000 tokens and the deeper four-stage pipeline had enough chunks to stay full. Decode, which must hold every active request’s key-value cache for the whole generation, ran on H20-141GB nodes and split again by objective, using single-node TP8 as a latency reference against PP2-TP8 for actual low-latency serving, and DP16-EP16 as an efficiency reference against DP32-EP32 for throughput and capacity.

Two techniques supplied most of the memory headroom. Humming MXFP4AFP8 compresses expert weights to MXFP4 with online FP8 activations, cutting the weight footprint on a chip with no native FP4 support. Online C128 replaces a per-index key-value compression scheme with a compact aggregate state, freeing more HBM for the cache itself. Combined, LMSYS reports the two techniques expand full-token KV capacity to 3.88 times the FP8 baseline on DP32-EP32 and 10.14 times on PP2-TP8.

On latency, the payoff showed up as speed rather than just headroom. Optimizing the DSpark speculative-decoding path cut peak decode latency, measured as time-per-output-token, by 74.8 to 78.0 percent at batch size 1 across context lengths from 8,000 to 1 million tokens. At the single-request extreme, the tuned H20-141GB configuration reached 271 output tokens per second against 383.7 tokens per second that LMSYS cites for a B300 reference, a 1.42x gap on hardware that trails B300 by roughly 45.6x on paper.

Prefill scaled in the opposite direction. The optimized path processes a 1 million token prompt in 43.7 seconds and sustains 8.45k input tokens per second per node at that length. The choice between prefill topologies is itself workload-dependent: PP2 cuts prefill latency by 16.7 to 19.5 percent versus PP4 at 4,000 and 32,000 token inputs, the two stay within 2 percent of each other through 64,000 tokens, and PP4 pulls ahead by roughly 26 to 45 percent as context grows from 128,000 tokens to 1 million.

The figures come entirely from LMSYS’s own benchmark harness, not an independent third party. The study’s acknowledgements also credit engineering teams from Ant Group as collaborators, but LMSYS does not specify how much of the routing and concurrency tuning reflects Ant Group’s own production traffic, a detail that matters for how directly these profiles transfer to a different workload mix. The accuracy check for the compressed weight path, a 95.5 percent exact-match score on GSM8K against a 95 percent internal threshold, is a single benchmark rather than a broad capability suite.

Teams serving large MoE models on memory-constrained or export-restricted GPUs should read this as a profiling template, not a configuration to copy outright. Measure the actual context-length and concurrency mix of the target workload before selecting a topology, because the binding constraint (compute, HBM capacity, or interconnect bandwidth) determines which profile wins at each operating point, and picking the wrong one shows up directly as wasted GPU-hours or missed latency targets.

LMSYS published this deployment engineering study on its blog on August 19, 2026.