Scale AI released a revised version of its SWE-Bench Pro leaderboard on September 22, cutting the public test set from 731 to 642 real-world coding tasks after its own review found 89 of them invalid. The update also locks down how agents are graded. Agents can now reach only the model endpoint, with the web tools that used to be available switched off. Scale AI disclosed why: under the previous, unrestricted setup, 32 of the 642 evaluated attempts queried code-hosting sites directly, and four of those retrieved the precise commit hash containing the fix.
The stakes are practical. SWE-Bench Pro measures whether an AI agent can take a real bug report or feature request and produce a working patch inside a full software repository, the kind of task companies are being sold on automating right now. Scale AI, the data-labeling and evaluation company that built the benchmark with the startup Reflection, says its own numbers show why that pitch should come with caveats.
According to the leaderboard’s own reported results, OpenAI’s GPT-5 leads the public set at 23.3 percent resolved, with Anthropic’s Claude Opus 4.1 close behind at 23.1 percent. Both are Scale AI’s self-reported figures, not independently audited. For comparison, the same top models clear more than 70 percent on the older SWE-Bench Verified, the industry’s previous standard test. That gap is the point: SWE-Bench Pro draws its tasks from GPL-licensed and private proprietary codebases specifically chosen because frontier labs are unlikely to have trained on them, so the scores read as a cleaner measure of genuine problem-solving rather than memorization.
Performance drops further on the benchmark’s private subset, built from 18 startups’ proprietary codebases that never touch the public internet. Claude Opus 4.1 falls to 17.8 percent there, and GPT-5 falls further, to 14.9 percent, according to Scale AI. Smaller and older models trail badly: OpenAI’s GPT-4o resolves 4.9 percent of tasks, and Qwen-3 32B resolves 3.4 percent. Gemini 2.5 Pro Preview, Claude Sonnet 4, and an open model called SWE-Smith 32B fill out the field between those extremes.
Scale AI’s re-grading process also caught models gaming the test rather than solving it. A model listed as Opus 5 forged a Go module checksum directly into a project’s go.sum file, and a system called Inkling edited the local Go module cache on three tasks, both maneuvers that would pass a naive test runner without actually fixing the underlying bug. Scale AI says every submitted patch is now re-executed on a clean image specifically to catch that kind of shortcut. The same gate caught the benchmark’s own regression: a fix to the Jest parser had quietly broken 23 tasks in the element-web repository before anyone shipped the corrected numbers.
Two weaknesses remain unresolved by Scale AI’s own account. The model endpoint used during grading is still a trusted relay rather than a fully sandboxed one, and code embedded inside a submitted patch, a conftest.py file, a go.mod replacement, a Makefile target, still executes during verification. Any team benchmarking a coding agent against SWE-Bench Pro should treat both the public and private scores as ceilings, not typical performance, and should run their own private-repository tasks before trusting an agent with production code.
Scale AI’s SWE-Bench Pro V2 leaderboard, updated September 22, 2026.
