The popular explanation for why language models do so well at math, that math answers are simple to verify, does not hold up under a closer look. That is the argument Steven Byrnes makes in a LessWrong essay published September 18.

Byrnes’s objection is straightforward. For anything beyond basic arithmetic, the thing doing the verifying is itself a language model. Saying math is “easy to verify” really just means language models are good at judging whether a math argument holds together. That explanation assumes the very skill it is trying to account for, so it does not actually explain anything.

His alternative theory rests on a different mechanism: what the model absorbed during pretraining. Byrnes estimates that a randomly chosen sentence from the published math literature is correct well over 99 percent of the time. A model trained mostly through imitation, predicting what comes next in text it has seen, will therefore end up mimicking mostly-correct reasoning by default. Only a smaller layer of curated mid-training data or reinforcement learning is then needed to sharpen how the model applies that reasoning.

Byrnes extends the same logic to programming. Code posted publicly tends to compile and run roughly as intended, so a model imitating that code inherits a similar baseline of “usually works.” The catch, in his account, is that public code is also frequently messy or inefficient, which is why labs have poured years of effort into RL and curated data specifically for coding, more than math required for a comparable payoff.

Outside math and code, Byrnes describes the published research in many fields as a mix of genuine insight and confident-sounding error, with nothing marking which is which. A model imitating that literature, he argues, will reproduce the same blend: real insight next to nonsense, indistinguishable on the surface. Fixing that through curation or RL, in his view, is a much harder problem than it has been for code.

The essay ties this directly to an active safety debate: whether an AI system could accelerate its own capability research, or automate parts of alignment research, once it becomes advanced enough. Byrnes frames the question as an empirical one rather than a philosophical one. It depends on whether the relevant research literature, machine learning ideas or new AI architectures, more closely resembles the reliability of published math or the noisier record found in most other fields. He does not answer that question himself, leaving it to readers to judge.

Byrnes revised the essay after publication, following pushback in the comments from a reader identified as Beren, who noted that some frontier labs already run reinforcement learning with verifiable rewards, RLVR, on math problems that have checkable right answers, such as the AIME competition set. Byrnes distinguishes that narrow case from the RL used on advanced, proof-based math, which he says still relies on an AI judge rather than a mechanical checker.

This is Byrnes’s own framework, offered without an accompanying experiment, and the underlying claim, that most math writing online is correct, is presented as an estimate rather than a measured figure. For teams betting product roadmaps on agents that reason their way through open-ended research problems, Byrnes’s framing suggests the more useful question to ask is not how good the model is at math, but how clean the training data looks for the exact task at hand.

Steven Byrnes, “Pretraining data, not verifiability, is why LLMs are especially good at math (and coding),” LessWrong, September 18, 2026.