A compiler that derives GPU kernels matches hand-tuned code in places

Cam Pedersen of Muni has published research on tang, a Rust compiler that generates GPU kernels from a model’s math instead of loading hand-written ones, and the results are mixed in an instructive way. Muni’s paper reports ties with llama.cpp and a few clear wins over PyTorch, alongside several slower results it does not hide.

The motivation is a maintenance bill. A modern model uses a few dozen operators, yet a framework ships thousands of kernels, because each operator needs a version per backend and each profitable fusion needs another. FlashAttention shows the cost: one hand rewrite for the forward pass, a second for the backward, a third for Hopper, and a fourth now arriving for Blackwell’s FP4 tensor cores. Muni argues each of these freezes a schedule that suits one shape on one chip.

Tang replaces the library with two constructs. Views are affine maps that describe how a computation reads its inputs (transposes, slices, broadcasts, padding). Comprehensions are reductions under a monoid, a combining rule that can be regrouped freely. Fusion is a single substitution rule, and flash attention falls out when the compiler recognizes online softmax as a monoid. No attention-specific code is involved. A gradient taken through a comprehension is itself a comprehension, which lets the identical compiler passes produce the backward pass. Muni says no intermediate tensors get written there on CUDA, WGSL or Metal.

One graph lowers to WGSL, CUDA tensor cores, Metal, OpenCL, GLSL and C, which covers everything from a datacenter GPU to the phone-class chip inside a humanoid robot. For a datacenter-versus-consumer comparison, Muni’s own numbers are the guide. Take an idle RTX 3090 running one transformer block. Derived code beats PyTorch (captured in a CUDA graph) by 1.2 times when the sequence is 512 tokens long and the block runs forward only. Cut the sequence to 128 and PyTorch wins by the same factor. Qwen3-8B decodes at 148.7 tokens per second as a single graph. That ties llama.cpp for short prompts and reaches 92 percent of its speed at 4,000 tokens.

The weak spots are specific. Derived attention forward trails FlashAttention-2 at every size tested, and on the RTX 3090 one configuration generates no tensor-core forward at all, leaving it roughly 150 times slower. The fused attention backward on CUDA computes in bf16, so it misses the f32 accuracy tolerance and the search rejects it. On Metal and Vulkan, the backward that avoids intermediates costs speed: forward plus backward runs at 0.68 to 0.93 times the standard schedule on Metal and 0.25 to 0.34 times under Vulkan.

The harder test is an 80-billion-parameter mixture-of-experts engine on an NVIDIA H200, where Muni replaced hand kernels with derived ones. The derived expert layer is 18 percent slower at batch 1 and 7 percent slower at batch 4. It ties at batch 8 and is 5 percent faster at batch 64. Inside the full model, decode steps take 5 to 12 percent less time at 16 to 48 concurrent streams, but the derived path scores 98.93 percent on the engine’s agreement check against a 99 percent bar. A whole derived decode step still trails the hand engine by 13 to 35 percent. Time to first token at 44,000 tokens drops from 8.33 to 5.54 seconds. Muni concedes the baseline is weak: running vLLM against the FP8 checkpoint gets a first token roughly 12 times sooner than the engine manages, so the improvement is measured against Muni’s own earlier code.

The paper also tries to say how fast a kernel could possibly run. A traffic bound and the best tile come from one linear program, with weak duality proved in Lean. Across 24 validated kernels on two GPUs, that floor stayed below every measured runtime. It sits close to measured time on the RTX 3090’s large kernels but only 0.05 to 0.66 of it on the H200, a gap Muni says it cannot yet explain.

The commercial angle is the hardware spread. Every new chip family forces teams to port or wait for kernel libraries, and a compiler that reaches within tens of percent of hand work on six targets changes that calculation, particularly for edge devices with no mature kernel ecosystem. Inference teams targeting more than one vendor should run tang’s lab on their own attention variants, but the numbers say to keep FlashAttention and vLLM where raw throughput on NVIDIA decides the contract.

Published by Muni on November 11, 2026, in research by Cam Pedersen.