Cerebras, the maker of dinner-plate-sized AI chips, published a walkthrough of how it runs models too big for any one of them. The company’s WSE-3T wafer holds 44 GB in on-chip memory, while a trillion-parameter model needs terabytes just for its weights. Cerebras says it splits such models across several wafers and still serves them at 1,000 to 2,000+ tokens per second.
The worked example is Kimi K3, an open source model with about 3 trillion parameters, built from 24 dense layers plus 69 of the linear-attention kind. Cerebras assumes 4-bit weights, an 8-bit key-value cache (the memory a model keeps of the conversation so far), and an average request of 256k tokens. Spread over 93 layers, each one comes to roughly 16 GB, small enough to sit whole on a 44 GB wafer.
Layer type decides how many share a chip. Dense layers store the keys and values of each token for each live request, so each gets a wafer to itself, with the leftover memory reserved for that cache. Linear-attention layers carry a compact, constant-size state per request, whatever the context length, so two fit on one wafer. That gives 35 wafers for the linear layers and 24 for the dense ones: 59 in total.
Because layers rarely need to be cut apart, the weights stay where they are. Only activations, the intermediate results of a calculation, hop between wafers, using RDMA over Ethernet at roughly 2 microseconds per hop. Cerebras compares the setup to an assembly line in which each station keeps its tools and the work moves past it.
The obvious objection is idle hardware. If one token must pass through every wafer in turn, most wafers would wait most of the time. Cerebras answers that it never sends a single request alone: while stage 3 handles a token for one request, stage 2 handles another request and stage 1 a third. Each stage can also take a micro-batch, meaning a handful of requests at once, so the load in flight is about stage count multiplied by micro-batch size. Shrinking the micro-batch speeds up each request, and growing it raises total throughput.
In its example each stage is one layer. The company says production runs use far finer slices of the model, and many more of them.
Concurrency is capped by memory. Each live request needs a private cache on every wafer that holds a dense layer, so the memory left after the weights decides how many requests the pipeline can hold. Cerebras walks through the arithmetic in its post but the post as published gives no final concurrency count for Kimi K3.
Everything here is Cerebras describing its own system. The post offers no independent measurement of the 1,000 to 2,000+ tokens per second figure, and it does not say which models or request mixes produced it. Cerebras itself says concurrency and batch size are not the best yardstick, and points instead to tokens per second per megawatt, measured at the same output speed. It publishes no such number here.
That omission is the useful signal for buyers. GPU clusters get compared on price per token and power draw, and Cerebras has just named the metric it wants to be judged on without filling it in. Anyone pricing inference for a multi-trillion-parameter model can send the company that exact question and compare the answer with their current cluster’s tokens per second per megawatt.
Reported by Cerebras in a company blog post on how it deploys trillion-parameter frontier models (no publication date given on the page).
