Two benchmarks that engineers routinely cite to argue one coding model beats another do not hold up once someone checks the methodology, according to a long essay by the software engineer and blogger Dan Luu. His subject is DeepSWE and Senior SWE-Bench, both used to rank how well AI coding agents handle real programming tasks.
The headline numbers already contradict each other. A casual read of DeepSWE has OpenAI’s prior-generation GPT-5.5 performing on par with Anthropic’s current Fable 5. A casual read of Senior SWE-Bench has Anthropic’s older Opus 4.8 beating OpenAI’s current GPT-5.6. Luu argues neither ranking matches what he and other working programmers actually experience from these models.
DeepSWE’s full test set is 113 tasks, each run four times and scored on a five-point pass scale. GPT-5.5 comes out ahead of Opus 4.8, 67 percent to 54 percent, tying on 34 tasks, winning 57, and losing 22. Luu’s objection is not the math. It is that almost none of the 79 tasks where the models diverge resemble the work he does day to day: only four are written in Rust, the language where he actually relies on coding agents, and he found none of those four representative of real tickets he has handled.
Senior SWE-Bench fares worse in his account. To count as a “tasteful solve,” an answer must clear a rubric score and stay within twice the length of a reference solution, a rule that turns a continuous quality judgment into a pass or fail cliff. Luu points to a task where GLM-5.2 scores “tasteful” at 121 lines against a 61-line reference: one additional line would have flipped the result to a fail. Another task’s reference solution is a single line of code, meaning the maximum length allowed for a passing answer is three lines total. Each condition on the benchmark is also run only once, despite what Luu says is greater variance between repeated runs of the same model than between different models entirely.
Luu says he consulted Aaron Levin, who worked on an evals team at Anthropic, to check his reasoning, and that Levin agreed with the conclusion. He frames that as reassurance rather than proof: the flaws he describes, tiny non-representative samples, single runs, and arbitrary scoring thresholds, are basic experimental-design problems, not specialist AI knowledge.
The piece lands as benchmark tables keep showing up in vendor marketing and in the model-selection debates that operators have on internal chat channels. Artificial Analysis, for one, updated its capability indices this month with Claude Fable 5.1 ahead in all six; Luu does not discuss those indices, but his checklist applies to any clean sweep like it. His broader point is procedural: a benchmark builder who reports a single summary score, on a task set too small to be representative and scored with hard cutoffs, can make almost any model look like the best one, and reversing the numbers on a rival benchmark is just as easy.
Teams choosing a coding model on the strength of a leaderboard screenshot should ask how many tasks the score covers, whether it was run more than once, and whether the tasks look anything like their own codebase before treating the ranking as a purchasing signal.
Dan Luu, danluu.com, September 2026.
