VIDRAFT publishes its full Gemma speed-tuning recipe, flag by flag

VIDRAFT, competing as vidraft-darwin, published the complete configuration behind its fastest verified entry in The Fast Gemma Challenge, a speed competition Hugging Face and the Google Gemma team ran on identical hardware. The submission served google/gemma-4-E4B-it on a single NVIDIA A10G at 510.58 tokens per second, with a perplexity score (PPL, where lower means less quality loss) of 2.3930, under the challenge’s roughly 2.42 ceiling.

Inference leaderboards usually stop at the score. Contestants rarely release the environment variables, kernel patches, and serving flags that produced the number, so nobody outside the team can tell whether a result reflects a real technique or undisclosed tuning hours. VIDRAFT posted its full manifest.json plus every serving patch file, which means a stranger can rerun the 510.58 figure today instead of taking it on faith. That transparency, more than the ranking, is the actual news: a leaderboard without published configs mostly measures who tuned longest, not who found something reproducible.

The hardware choice matters as much as the software. An A10G is a mid-tier accelerator that costs a fraction of an H100 and sits in far more production inference fleets, which is why a result on it travels to teams without frontier compute budgets. It also means these specific numbers, tuned around one GPU’s memory bandwidth and kernel behavior, will not simply carry over to an H100 or a different accelerator generation.

“Verified” here has a specific meaning worth stating plainly. Hugging Face and the Google Gemma team, as challenge organizers, re-run each submission’s config against a private prompt set and only mark it VERIFIED if the throughput and PPL both hold. That is organizer verification inside a single competition, not an independent third-party audit. It matters because a separate, unverified VIDRAFT-adjacent run hit 535.91 tokens per second, faster than the winning entry, but its PPL landed above 2.44 and it failed the quality bar. VIDRAFT’s own framing: not the fastest run overall, the fastest among the ones that verified.

VIDRAFT did not publish an unoptimized baseline figure, so the total speedup this stack delivers over a stock vLLM deployment cannot be calculated from what it released. Only one individual optimization got a quantified number: a synthetic warmup phase that fires 64 one-token prompts before the timed run to finish CUDA-graph capture and JIT compilation in advance. Skipping it cost roughly 15 tokens per second, according to VIDRAFT.

The rest of the stack is described but not broken into individual tokens-per-second contributions. A sliding-window attention limit of 188 tokens targets the KV-cache memory bandwidth that VIDRAFT identifies as the decode bottleneck; the team says 128 tokens hurt PPL and wider windows slowed decoding, making roughly 188 to 192 the workable range. A kernel tuning value it calls CTK49 was swept across 44, 48, and 49 before settling on the highest setting that stayed inside the PPL budget. Speculative decoding, a drafter model predicting multiple tokens per step, adds throughput without its own reported figure. Precaching was switched off deliberately: VIDRAFT says a precache path can inflate the number a team measures on its own hardware while failing the organizers’ private re-run, so leaving it off keeps the self-reported score honest.

The governing rule VIDRAFT states outright is that any change moving PPL got dropped, regardless of the speed it added. That discipline, not any single flag, is what separates a verified 510.58 from an unverified 535.91.

Teams serving similarly sized models on A10G-class hardware can copy the sliding-window and precache settings directly and test them against their own workload. The speculative-decoding and pruned lm-head pieces depend on drafter weights and keepsets VIDRAFT built from community-shared assets, so reproducing those requires sourcing or training equivalents rather than flipping a flag.

VIDRAFT detailed its verified Fast Gemma Challenge submission and full configuration file in a post on the Hugging Face blog under the FINAL-Bench organization.