OpenAI reviewed SWE-Bench Pro, the benchmark most coding-agent labs cite when announcing new models, and concluded that roughly 30 percent of its public tasks are broken. The company reached that figure by examining how the tasks were constructed, where models failed against them, and what the task metadata actually recorded. A benchmark this compromised means a meaningful share of the coding-ability claims built on it over the past year deserve a second look.
The audit sorted the broken tasks into three failure modes. Some tasks are unsolvable regardless of a model’s ability, meaning no correct submission exists that the grading harness would accept. Others carry incorrect labels for what counts as a passing solution, so a model can solve the underlying problem and still fail the score. A third group breaks because of environment setup errors unrelated to coding skill at all, such as dependency or configuration failures that block any submission from running.
This is not the first time a lab has had to fix SWE-Bench’s foundations. OpenAI released SWE-bench Verified in August 2024, a human-screened subset of the original SWE-bench, after researchers found the source benchmark contained tasks that could be solved by luck and test harnesses too weak to reliably separate real fixes from lucky guesses. The new audit repeats that same pattern on SWE-Bench Pro, a version built specifically to be harder and more resistant to memorization. That the newer, harder benchmark still needed this kind of scrutiny says the underlying construction problem was never fully solved, only pushed forward.
The tension in this audit is that OpenAI is both the auditor and a repeat citer of SWE-Bench class scores in its own model launches. Labs report benchmark numbers from their own eval runs rather than from a shared, independently verified task set, which is exactly the condition that lets a 30 percent broken-task rate persist for as long as it apparently has. No frontier lab has an incentive to flag noise in a benchmark that makes its own model look strong.
SWE-Bench Pro results also feed into safety and autonomy evaluations, not just capability marketing. Labs use coding-agent benchmarks as a proxy for how much unsupervised work a model can be trusted to do, including assessments tied to autonomous software risk. If close to a third of the underlying tasks are miscounting model performance for reasons that have nothing to do with skill, the safety conclusions drawn from those same scores are standing on the same uncertain ground.
Anyone citing a SWE-Bench Pro leaderboard number in a vendor comparison, an investment memo, or a safety review should ask which specific tasks a model passed rather than trusting the aggregate percentage, at least until the benchmark’s maintainers publish a cleaned and independently verified task set.
OpenAI detailed the findings in a post on its company blog on July 9, 2026.
