Eight researchers, with Jingwei Li as the submitting author, published a pretraining study that turns “how many times can I reuse my best data” into a number labs can actually calculate. Their finding: the ideal repeat count for a domain of high-quality data rises slightly as a model gets bigger, and that count can be predicted from a much smaller, cheaper model trained under the same conditions. For labs that have already scraped the open web clean, this is a budgeting tool, not a curiosity.
The problem the paper addresses is structural. Every time a lab trains a larger model, it also has to feed it more tokens to keep a healthy tokens-per-parameter ratio (TPP), the standard measure of how much data a model sees relative to its parameter count. General web text scales along with that growth. Specialized, high-quality data, the kind that teaches a model math, code, or a technical domain well, does not. As models and their token budgets grow, that scarce material becomes a shrinking slice of the training mix unless a lab repeats it.
Repetition is the obvious fix, and the obvious risk: repeat a dataset too many times and the model starts memorizing rather than generalizing, a failure mode researchers call overfitting. The authors set out to quantify the trade-off under conditions that mirror how labs actually scale training runs today, where the token budget grows in step with model size rather than staying fixed.
Three results stand out in the abstract. First, holding TPP constant, the optimal number of repeats for a given data domain increases mildly as model size goes up, a direction the authors describe as counterintuitive going in. Second, across different data domains, the ideal repeat count tracks closely with how low a domain’s validation loss gets: domains the model already fits well can absorb more repetition than domains it struggles with. Third, and this is the operationally useful part, how much unique data exists in a domain barely moves the ideal repeat count at all. Data volume, in other words, is a weak predictor. Loss behavior is the strong one.
That third finding is what makes the approach practical rather than purely descriptive. If the right repetition schedule depended mainly on model scale or dataset size, a lab would need to run expensive experiments at or near production scale to find it. Because the paper’s central claim is that repeat counts tuned on small proxy models, at the same TPP, transfer usefully to much larger models, a team can run the search on a fraction of the compute and carry the answer upward.
Who this helps is anyone training frontier-scale models on a mix of abundant general text and scarce specialized corpora: code, math, scientific literature, or any vertical dataset that will not grow at the same rate as the model. Who it does not help, at least based on what the abstract establishes, is anyone whose domain data is already effectively unlimited, or anyone operating far outside the proportional token-to-parameter scaling regime the study assumes. The paper does not report the actual repeat-count numbers, benchmark deltas, or which domains were tested, so treat this as a validated methodology rather than a ready-made table.
Teams currently guessing at repetition schedules by running full-scale ablations should test whether a small proxy model, matched on TPP, reproduces their production-scale results before committing more compute to the search.
Findings are drawn from the arXiv preprint “Scaling Domain Data Repetition in LLM Pretraining” by Jingwei Li, Xinran Gu, Rui Dai, Xintong Hao, Chengyin Xu, Yan Wu, Shuran Zheng, and Jingzhao Zhang, submitted August 14, 2026.
