Dmitrii Zatona, an engineer who writes at zatona.dev, published an analysis on September 6 arguing that a shared benchmark name tells a reader almost nothing about whether two scores can be subtracted from each other. His example takes a single model family and sets two of its builds side by side, both filed under the same mmlu label, the same accuracy metric, the same fractional unit. Build 42 scored 0.781. Build 44 scored 0.79. Feed that pair into the verification tool he built, apl-ai-eval, and it returns a single word: incomparable.
The reason is not a rounding dispute. Zatona traces the gap to the evaluation records attached to each score, and those records disagree on nearly everything except the benchmark’s name.
Start with which questions were asked. The original MMLU paper defines a 14,079-question test set and a separate 285-question development set meant only as a source of few-shot examples. Build 42’s record scores on that 285-question dev set. Build 44’s record scores on something the crate’s own test data calls “test-lite,” a label that, per Zatona’s search of Hugging Face and GitHub, corresponds to no published dataset. Two runs, two different question pools, one shared name.
Then there is how each answer got graded. Build 42 used exact string matching against the correct letter. Build 44 used a language model as judge. Zatona cites a 2023 paper on LLM-as-judge reliability showing that even the most self-consistent judge model agreed with itself only 65 percent of the time across repeated trials, with weaker judges falling to 46 and 24 percent. Swap a string-matching grader for a judge model and the score stops measuring the same thing, no matter what the benchmark is called.
Formatting compounds both problems. Zatona points to Anthropic’s own 2023 writeup showing that answer-option punctuation alone can shift MMLU accuracy by about five percentage points, and to a separate study finding that simply moving the correct answer to a different letter position swung one model’s score from 67.2 down to 60.9 while lifting another’s by more than 15 points, enough to flip which model ranked higher. None of that touches a model’s actual capability. It only touches how the question was presented.
AI Insiders reported on September 8 that OpenAI’s AGI claim rested on a score produced by a provider-specific adapter rather than the field’s standard harness, and that the identical model measured 62.7 percent under one scaffold and 99.9 percent under the other. Zatona’s argument generalizes that incident. The scaffold problem was not a one-off configuration bug; it is what happens whenever two evaluators agree on a benchmark’s name and nothing else.
Zatona’s framing borrows from measurement science rather than machine learning. Comparability, he argues, is a property of the reference two results trace back to, not a property of the numbers themselves, the same logic that keeps a clinical lab from treating two calibration standards as interchangeable just because both report the same unit. His tool enforces that logic by hashing each evaluation’s full procedure, the runner, the grader, the question split and the prompt format, into a single record, and it refuses to compute a difference between two claims unless their records match or an explicit bridge translates between them.
The practical upshot for anyone reading a model comparison chart: a leaderboard row is only comparable to another row when both were produced under the same harness, the same split and the same grading procedure, which means most published side-by-side model comparisons are not actually comparisons. Before citing a benchmark delta in a procurement decision or a public claim, ask the vendor for the runner ID, the grader ID and the dataset split, not just the benchmark’s name.
Dmitrii Zatona published this analysis at zatona.dev on September 6, 2026.
