Independent researcher Giles Thomas set out to train a language model from scratch, matching the architecture of OpenAI’s original GPT-2 small. When he compared his finished models against OpenAI’s 2019 weights, the newer models lost on the metric that was supposed to matter most.
Thomas ran two evaluations. The first was a straightforward cross entropy loss test on a held-back set of web text sequences, a measure of how well a model predicts the next token. The second, which he calls the IFT eval, fine-tuned each model on samples from the Alpaca instruction-following dataset, then had another model, GPT 5.5, score the resulting answers out of 100 against a held-back test set.
On the technical measure, several of Thomas’s models beat OpenAI’s small GPT-2 weights outright. Two of his JAX-trained runs posted lower test loss, 3.418784 and 3.420089, than OpenAI’s small weights at 3.499677. On the instruction-following measure, the result flipped hard. OpenAI’s small weights scored 26.73 out of 100 and ranked second overall, beating Thomas’s best model on that eval, “Cloud FineWeb, 8x A100 40 GiB,” which scored 20.71. OpenAI’s medium weights, roughly double the parameter count of the small models, topped the table at 41.62.
The gap showed up in training behavior too. OpenAI’s weights needed only two epochs of fine-tuning before their validation loss began climbing, the signal that a model is starting to overfit. Thomas’s own models needed anywhere from three to seven epochs to hit that same point, a sign they started further from a workable instruction-following solution.
Thomas frames the discrepancy through what he calls the loss landscape: the idea that pretraining and fine-tuning optimize against related but different surfaces, and that a model which lands in a good spot for one is not guaranteed to sit near a good spot for the other. By that logic, OpenAI’s pretraining run happened to land somewhere better positioned for instruction fine-tuning, even though it was trained on a completely different web-text corpus than his own.
That last detail matters. Thomas’s test set was drawn from the same FineWeb corpus his models trained on, while OpenAI’s weights were built on a private WebText dataset compiled from Reddit outbound links that received at least three karma, according to the original GPT-2 paper. Despite that mismatch, OpenAI’s small weights still beat nearly all of his PyTorch models on his own test set. Thomas argues that is the more telling result. A model trained on unfamiliar data outperforming most of the models trained specifically for that data suggests OpenAI’s weights are simply stronger.
He has already ruled out one explanation. A JAX run trained with dropout, a regularization technique some researchers credit with improving generalization, scored 12.94 on the IFT eval, among the worst results in the entire table. Data quality remains an open question: OpenAI’s Reddit-filtered WebText may or may not beat FineWeb on quality, and Thomas notes that his models trained on the more heavily curated FineWeb-Edu subset scored worse on raw test loss but relatively better on instructions, which complicates any simple story.
His leading hypothesis heading into the next post is overtraining. Thomas trained his own models on 3.2 billion tokens, the Chinchilla-optimal ratio of roughly 20 tokens per parameter that became standard after a 2022 research paper. GPT-2 predates that heuristic by three years, and Thomas suspects OpenAI trained past that ratio, over more epochs, than his own runs did. He plans to test the theory on a newly built dedicated training machine he calls poppy.
Thomas has not identified the cause, and he says so directly: this is the first post in a planned series, and the explanation is still unconfirmed. The pattern he has already documented stands on its own as a caution for anyone selecting a model by benchmark. Loss and downstream usefulness diverged in a controlled, small-scale setting with numbers to show for it, the same failure mode that shows up whenever a team picks a production model purely on a leaderboard score. Teams evaluating models for user-facing instruction tasks should weight held-out behavioral tests over raw perplexity comparisons until Thomas’s series identifies what is actually driving the gap.
Giles Thomas detailed the results on his personal blog on July 29, 2026.
