Cohere built a serving system around what it calls a decode megakernel, and on its own model, North Mini Code, the approach reaches 292 tokens per second at batch size 1, a claimed 1.58 times the throughput of the open-source serving stack vLLM. Cohere published the design and the benchmark numbers on its engineering blog. The company also reports that the speed gain holds as batch size grows and out to 256,000 tokens of context.
The idea underneath the name is straightforward once it is unpacked. Normal LLM serving software turns a single decoded token into a chain of separate GPU programs, one for each math step: project the query and key vectors, run attention, run the routing network, and so on, waiting for each program to finish before the next one starts. Every one of those handoffs costs fixed overhead, and the chip’s memory controller often sits idle in the gap. Cohere’s megakernel collapses that whole chain into one long-running program that stays resident on the GPU for the full step, so the pauses between operations mostly disappear.
Why that matters comes down to what actually limits a decode step. Generating one token means pulling nearly the full weight file of the active parameters out of memory, and doing comparatively little arithmetic with it. North Mini Code is a 30-billion-parameter model that activates 3.3 billion parameters per token, so each step moves roughly 6.6 gigabytes of weights in 16-bit precision, plus about half a gigabyte of cached attention state at an 8,000-token context. An H100 GPU can move 3.35 terabytes per second through its memory. Divide one by the other and you get the theoretical ceiling, what Cohere calls speed of light: about 470 tokens per second if the chip did nothing but shuttle those bytes with zero waste.
That ceiling is why the comparison numbers need a second look before anyone treats them as settled. Cohere says vLLM serves North Mini Code at 185 tokens per second, 39 percent of that ceiling, while its megakernel hits 292 tokens per second, 62 percent. The gap between the two techniques is real, but both remain well short of the physical limit, and the 470-token figure is specific to one model’s parameter count, one precision format and one GPU generation; change any of those and the ceiling moves. Cohere frames the full production system, which additionally has to hold many requests in flight at once, keep the attention cache in pages rather than one contiguous block, and serve wildly uneven sequence lengths together, at 1.25 to 1.41 times vLLM end to end, below the 1.58 times that covers decode by itself.
None of this is independently verified. Cohere is measuring its own system against a baseline it configured, on hardware and a model of its choosing, and it says the result loses no measurable accuracy, without publishing a third-party audit of that claim.
The bigger point survives the caveats. Inference is now the line item that determines whether serving a model turns a profit, and a 1.58 times throughput gain that comes purely from software, with no new chip and no smaller model, is worth more to an operator’s margins than most incremental model upgrades. It is also the kind of advantage that tends to compress once the system it was measured against ships an update of its own; vLLM’s maintainers have closed similar gaps before. Teams running open-weight models at scale should treat this as a technique worth testing against their own workload, not as a fixed multiplier to bank on.
Cohere published these benchmark figures and the megakernel design on its engineering blog, in a post bylined Xiaochun Tong, Conway Zhu and Donglu Wang.
