A new preprint puts a number on a question the transformer architecture has never directly answered: what do its feed-forward layers actually buy. Every block in a standard transformer alternates two components: an attention layer that mixes information across tokens, and a feed-forward network (FFN) that processes each token’s representation on its own. Interpretability researchers generally treat the FFN as where a model stores factual knowledge, and it eats roughly two out of every three parameters outside the embedding table. A preprint led by researcher Henry Ndubuaku, posted to arXiv on July 20, ran the obvious experiment nobody had done cleanly: delete the FFN and measure what breaks, holding parameter count, compute, and depth fixed in turn.
The authors trained small attention-only decoder models, which they call Simple Attention Networks, against ordinary transformers under three separate matching conditions: equal parameter count, equal compute, and equal depth, spanning 2 to 48 layers and up to 105 billion training tokens. Model sizes ranged from 6 million to 87 million parameters, tiny by frontier standards, the tradeoff a study like this makes to afford enough controlled comparisons.
Deleting the feed-forward layers outright, without adjusting anything else, was expensive: loss rose 0.47 nats at equal depth and 0.26 nats at equal compute. Nats measure prediction error on a logarithmic scale, so that gap is substantial for models this small. Letting the attention-only model spend its freed parameters on more depth instead erased nearly all of it. At matched parameter count the gap fell to 0.006 nats, about a quarter of one percent of the loss, reproducing to roughly 1 part in 10,000 across repeated runs. It narrowed further from 5 billion tokens to 105 billion, and held near 0.02 nats across a 29-fold range of model sizes.
Three separate measurements traced the remaining gap to a single cause: parametric recall, meaning facts a model has to retrieve from its weights rather than pull from the prompt in front of it. The attention-only models matched or beat standard transformers when the answer lived in the context window, and lost ground where the answer had to come from memorized knowledge. Weight analysis backs this up: the matrices that route attention settle into their final structure early in training, while the matrices that store content build up rank slowly, and removing the FFN pushes that accumulation into the attention output layer instead. A technique called QK-normalization, not the feed-forward layers, is what the paper credits with keeping 48-layer attention-only stacks trainable. A pre-registered follow-up held up: the authors said in advance that knowledge-dense text would open a gap of 0.02 to 0.05 nats, and a matched pair trained on the fineweb-edu corpus came in at 0.040.
The result matters beyond interpretability circles. Feed-forward layers hold roughly two thirds of a decoder’s non-embedding parameters, so they also account for a comparable share of memory footprint and serving cost in production. Moving a meaningful slice of that budget into attention depth for a fraction of a percent of loss is a claim about economics, not just architecture. This is a single preprint, run at parameter counts far below any deployed frontier model, and it has not been peer reviewed or replicated by an outside lab. The gap it does find, small but reproducible and concentrated entirely in memorized-fact recall, needs to hold at billion-parameter scale before it changes how anyone builds a production model.
Anyone budgeting parameters for a new pretraining run now has a testable hypothesis: trading feed-forward capacity for attention depth may recover most of the lost accuracy, at least at the scales this paper covers. Confirming that at production scale, not at 87 million parameters, is the experiment someone needs before it changes a serving bill.
Findings from a preprint by Henry Ndubuaku, submitted to arXiv on July 20, 2026.
