Every model on Cognition’s new coding leaderboard scored lower than it did on the previous version, and the best of them, Claude Opus 5.5, reached 45.5 percent, down from 54.6. Cognition, the company behind the Devin coding agent, published FrontierCode 2.0 on its blog on 27 October. The first FrontierCode, which we covered in June, asked whether a maintainer would merge a model’s code. The sequel makes that harder to pass.

The drop gets steeper as you go down the list. Gemini 4 Argon went from 47.5 percent to 26.3, DeepSeek V4.1 Flash from 43.9 to 19.3, and Kimi K3 from 44.2 to 17.7. On version 1.1 the eight models Cognition compares sat within about eleven points of each other. On 2.0 the gap between first and last is roughly 28 points, which is the separation Cognition says it wanted.

Three changes explain most of it, according to Cognition. Tasks come from 55 percent more repositories and now include security patches and performance work. The reference solutions touch 67 percent more files and 89 percent more lines than in 1.1, in codebases 25 percent larger. And there are 43 percent more blocking criteria, the checks where a single failure sets the whole task to zero.

The category that grew fastest is “thoroughness,” which Cognition defines as things a capable engineer would infer without being told. Say a prompt points at one broken function while its twin next door has the identical flaw: the thorough agent repairs both. Blocking checks of this kind went from 168 to 501, and they are also where models fail most: 19.2 percent of thoroughness checks failed on 2.0, against 11.7 percent on 1.1. At every reasoning-effort setting Cognition tested, thoroughness had the highest failure rate of any category.

Cognition also reversed itself on one rule. Version 1.0 had scope checks that penalized models for changing more than the prompt asked, and the company says that was controversial, since some of the failing solutions were reasonable improvements. In 2.0 it loosened them, and the scope failure rate fell from 9.0 percent to 3.8. Scores still fell across the board, so the harsher results cannot be blamed on stricter scope grading.

That scope history also explains an oddity: on 1.1, models sometimes scored lower when given more reasoning effort. Cognition found that more effort pushed models to produce bigger diffs spread over extra files. On the earlier test, Opus 5 passed 69.2 percent of scope checks at low effort and 51.1 percent at maximum effort. Cognition says it also saw more cheating at high effort, with models “accidentally” pulling the solution commit, and some cases of a model talking itself into a misreading of the task.

On price, Cognition reports that Opus 5.5 holds the high end of the score-versus-cost chart, while Luna, Haiku, and Sol sit at the cheap end. GPT-6.1 Sol gets particular credit for its score relative to cost. Open-source models, the company says, have significant room to improve, western ones most of all.

Cognition builds and grades this benchmark, and it sells Devin. The post, as published, does not give a Devin score on 2.0. It does describe an audit in which an agent found a security flaw in an upstream repository’s own code, which the team reported to the maintainers after human review. In 80 percent of tasks the reference solution now differs from the upstream pull request, up from 60 percent.

A coding-agent buyer who picked a model from a score near 50 percent should treat that number as retired. Cognition has not said where Devin lands on the new test, and that is the figure the market will want next.

Reported by Cognition on its blog, 27 October 2026.