Vercel published DeepsecBench, a leaderboard that scores language models on how well they catch real vulnerabilities in application code, testing each one against a codebase whose flaws are already documented. OpenAI’s GPT-5.6 Sol finished first, but even at its top reasoning setting it caught under a third of the confirmed bugs reviewers had already found. That gap between a benchmark’s top rank and its actual detection rate matters for any team deciding whether an AI model can replace, rather than supplement, a human security review.

On its xhigh reasoning setting, GPT-5.6 Sol scored 35.58 out of 100, with 30.7 percent recall and 96.3 percent precision, at a cost of $55.98 and a run time of three hours and 39 minutes. Claude Opus 5, run on Anthropic’s max setting, placed second at 32.57, catching 28.1 percent of known flaws at 88.0 percent precision, but it was the priciest entry on the board at $127.93 for two hours and 34 minutes. Dialed down to Anthropic’s medium setting, Opus 5 still placed third with a score of 28.36, recall of 24.7 percent, and precision of 70.5 percent, in under an hour and at roughly a quarter of the max setting’s cost.

Cheaper models closed much of the gap. Kimi K3, from Moonshot AI, scored 17.56 on its high setting, with 14.7 percent recall and 77.1 percent precision, for $12.38 and just under two hours of runtime. Grok 4.5, xAI’s model, scored 15.58 at high, with 13.0 percent recall and 77.8 percent precision, for $5.60 in one hour and 24 minutes, the cheapest run in Vercel’s featured comparison. Turning GPT-5.6 Sol’s reasoning down to medium dropped its score to 25.10, with recall of 21.2 percent and precision of 94.3 percent, but cut the cost to $17.95 and the run time to about 31 minutes, roughly 71 percent of the top score for a fraction of the price.

Notably absent from the board is Fable 5, Anthropic’s most capable model, which Vercel says declines security work entirely, including defensive scans. Vercel also flags that its sticker prices understate real-world cost: the benchmark scans only 50 files, while a production codebase can run roughly 100 times larger, pushing a full Kimi K3 sweep toward $1,200 and a full GPT-5.6 Sol xhigh pass above $5,000.

DeepsecBench runs against an open-source codebase captured at a commit just before a large batch of vulnerabilities got fixed, checking 50 entry-point files against 231 human-verified findings and scoring each run three times before publishing the median. Vercel weights recall twice as heavily as precision in its formula, reasoning that a missed vulnerability stays exploitable while a false positive only costs review time. The company keeps the underlying repository, commit, and file list secret so a model cannot simply have memorized the eventual patches. Vercel says a model doing that would score near-perfect recall, and instead the best run reached only 30.7 percent, with 20 of 25 runs finishing under 20 percent.

That secrecy is also the benchmark’s central weakness. No outside researcher can inspect the 231-item golden set, audit the judge model that grades borderline findings, or reproduce Vercel’s numbers, so the ranking has to be taken on the company’s word rather than checked independently.

For a team building an AI-assisted security review pipeline, the practical read is to match model tier to task: run a cheap, fast model like Grok 4.5 on every merge, and reserve an xhigh-tier pass for milestone releases rather than paying frontier prices on every pull request. Given that even DeepsecBench’s top performer missed roughly seven in ten confirmed bugs, no model on this list should be the last check before code ships.

Vercel detailed these DeepsecBench results, including its full per-model leaderboard, in a post on the Vercel engineering blog.