Dwarkesh Patel and Jerry Han published an analysis on Dwarkesh Patel’s Substack (dwarkesh.com) arguing that data quality, not architecture, explains most of the compute-efficiency gains in pretraining between 2019 and 2025. Their headline number: 3.24 times more of the gain traces to data than to model changes. The stakes are practical, not academic. If a lab wants more capability per dollar of compute, the analysis says the biggest lever has been what you feed the model, not how you build it.
The method is the story here, because the 3.24x figure is unverifiable without it. Patel and Han took a published, open model recipe from each year (GPT-2 in 2019, OLMo-2 in 2025, with public releases in between) and paired each one with that year’s public data corpus (OpenWebText in 2019, UltraFineWeb by 2025). They then trained every recipe against every corpus, crossed in a grid, at five compute budgets up to 1e19 FLOPs. Because the datasets differ, comparing raw training loss would be meaningless, so they scored each run on OLMES, a bundle of ten easier multiple-choice benchmarks, and converted the score gaps into compute multipliers: how much less compute a later recipe or corpus needs to match a 2019 baseline’s capability. At the top compute budget, data corpus swaps bought 12.0x efficiency; model recipe swaps bought 3.7x. The ratio is 3.24.
The more consequential claim sits underneath that ratio. Patel and Han report that gains from data and gains from model changes are largely additive rather than compounding: a linear regression crossing model effect and data effect against the observed OLMES scores explained 88 percent of the variance, leaving roughly 12 percent for interaction effects and evaluation noise. That is their finding, built on a linear model of a fairly small experimental grid, not a proof that architecture and data can never interact at other scales or tasks. Worth flagging plainly: it rests on the assumption that a linear additive model is the right lens, and the authors ran this at compute budgets far below what frontier labs train on today.
Patel and Han are also careful about what the model side actually bought. They argue its main contribution was not raw efficiency but making larger training runs survivable at all: stability tricks, mixture-of-experts routing, sparse attention, and kernel-level tools like FlashAttention exist mostly to stop bigger runs from breaking, not to squeeze more out of a fixed compute budget. And the data advantage may shrink at the frontier. Small models benefit heavily from curation because they lack the capacity to absorb noise, while oversized frontier models, deliberately trained on more tokens than would be loss-optimal so inference stays cheap, seem to do better absorbing large noisy corpora than aggressively filtered smaller ones.
That produces a real fork for operators. A small lab training compact models should treat its data pipeline as the primary lever, ahead of architecture tweaks. A frontier lab pushing scale should expect diminishing returns from further curation and look instead at synthetic data generation, since natural web text is a finite, largely exhausted resource. There is also a broader implication the piece does not spell out directly: if most of the visible six years of pretraining progress ran through data engineering rather than published architecture papers, then the architecture literature has been a weak proxy for what actually moved capability, because data pipelines are the part labs describe least.
Operators building or evaluating small models should ask any vendor which lever, data or architecture, produced a claimed efficiency gain, since the two do not automatically transfer between different scales.
Dwarkesh Patel and Jerry Han, published on dwarkesh.com on 8 September 2026.
