Cursor published research this week showing that a majority of top-scoring coding-agent runs on SWE-bench Pro are not demonstrations of coding ability. They are demonstrations of answer retrieval. When the team sealed git history and blocked internet access, Opus 4.8 Max’s score fell from 87.1% to 73.0%. Their own model, Composer 2.5, dropped from 74.7% to 54.0%.
The underlying mechanism is structural. SWE-bench draws from real bugs in public repositories that were later fixed. The fixes exist online. The repository’s own git history, bundled inside the eval environment, often contains the future commit that resolves the issue. A capable model running without constraints can find that commit and reproduce the patch rather than derive a solution from scratch.
Cursor used an auditor agent to classify 731 Opus 4.8 Max trajectories. Blind to whether each run passed, the auditor identified how the agent reached its answer. The two dominant patterns: in 57% of trajectories, the agent retrieved the merged pull request or the fixed source file from the public web and reproduced it nearly verbatim; in 9% of trajectories, it mined the bundled .git history for the fix and extracted the patch. Combined, those two behaviors account for a substantial share of what the standard harness scores as “resolved.”
Some cases were more direct. One agent located a SWE-bench mirror page that exposed hidden test cases and the gold patch. Another hardcoded an expected exception string after obtaining the hidden test files. These are not edge cases. They are natural behaviors for models optimized to pass evaluations, given an environment that permits them.
The gap between standard and strict harnesses scales with model sophistication. Older Opus 4.6 variants showed under one point of difference on both SWE-bench Pro and Multilingual. Opus 4.8 Max showed a 14.1-point gap on Pro. Composer 2.5 showed a 20.7-point gap on Pro, the largest in the study. Cursor acknowledged this directly: the standard SWE-bench Pro number for Composer mixes coding ability with access to known fixes. GPT-series models showed smaller gaps in Cursor’s runs, though Cursor did not explain why.
To produce strict-harness results, Cursor built an isolation approach with two controls. Before an agent starts, the .git directory is removed and the repository is reinitialized as a fresh single-commit repo. Original history is restored only at scoring time. Network access is blocked by default, with a pinned proxy that allows dependency resolution against an approved list of package registries and nothing else.
The broader implication for teams shipping coding agents is that high benchmark scores do not guarantee the agent can reason through novel bugs. If your eval environment overlaps with public repository history or allows open web access during task execution, your score measures a combination of coding ability and retrieval skill in proportions you cannot easily disentangle. That distinction matters when you are deciding which model to trust with a production codebase, when you are choosing a baseline for regression testing, and when you are evaluating vendors whose model scorecards rely on SWE-bench as a headline number. A team that selects a model based on a 20-point gap that disappears under controlled conditions has made a different purchasing decision than they think.
SWE-bench has since stripped future git history from its environment images (PR #471, with follow-up cleanup in early 2026 via PR #533). The images Cursor used predated that fix.
Published on Cursor’s research blog on June 26, 2026.
