Specific, the AI evaluation company behind withspecific.com, published a new coding benchmark called Real-SWE built entirely from private, licensed production codebases. The top score, a 38.8% resolution rate from Fable 5.1 running inside the Claude Code harness, cannot be independently checked by anyone outside Specific, because the underlying repositories are not public.
That is the trade Specific is explicitly making. Public suites like SWE-bench draw tasks from open-source repositories, which means any model’s training data may already contain the answer. Specific licensed codebases from three real companies instead: a consumer app with more than 50,000 users and a top 100 App Store ranking, a fintech platform processing more than 100,000 bank statements, and an enterprise sales platform handling complex business workflows. According to the company, the arrangement matters because “99% of tokens in real-world enterprises are hidden away from the frontier models,” so tasks drawn from them are unlikely to have been trained on already. Only a small sample of eight tasks is available to outside researchers who request access; the rest of the benchmark stays behind Specific’s own wall.
The full leaderboard, evaluated as pass@1 averaged over eight runs per task, is worth reading in order rather than by headline number alone. Fable 5.1 topped it at 38.8% inside Claude Code. GPT-6 Astra followed at 33.8% inside Codex CLI, then Gemini 3.8 Flash at 31.2% inside Gemini CLI, GLM 5.3 at 28.8% inside Claude Code, Grok 4.6 and Muse Spark 1.3 tied at 23.8% inside Grok Build and Muse Code respectively, Kimi K3 at 18.8% inside Kimi Code, and GPT-5.6 Sol at 16.2% inside Codex CLI. Specific evaluated model-and-harness combinations together rather than models alone, on the reasoning that enterprise engineers work through a specific tool, not a bare model.
Cost tells a different story than rank. Fable 5.1’s winning run averaged $6.96 per rollout, the most expensive model in the field and nearly three times Gemini 3.8 Flash’s $2.50. GPT-6 Astra’s second-place run cost $4.67, and GLM 5.3’s fourth-place run cost $5.12. The two cheapest models, Gemini 3.8 Flash at $2.50 and GPT-5.6 Sol at $2.65, landed third and last. Specific did not frame this as an efficiency finding, but any team reading the leaderboard as “best model” rather than “best model per dollar” is answering a different question than the data supports.
The harder tasks expose the gap further. Specific’s own breakdown shows that on tasks its runners took ten minutes or longer to resolve, 398 of 542 attempted rollouts failed, a 73.4% failure rate across the full model set. Six of ten sampled tasks scored resolution rates below 15%. Specific attributes the weakness to company-specific engineering patterns: models are “weaker at understanding company coding patterns and frequently miss requirements or don’t verify” against them, according to the company’s own writeup. That claim, like the leaderboard itself, is Specific’s characterization of its own private data, not something a third party can verify against the source repositories.
The company did not disclose in this release what it sells or how it plans to commercialize the benchmark, but the incentive is worth naming anyway: Specific controls which private codebases get licensed, which tasks get sampled, and which eight get released for outside scrutiny. A benchmark showing every frontier model failing on realistic enterprise code is also an advertisement for the idea that public leaderboards overstate what these models can do, a framing that benefits whoever positions themselves as the evaluator of record. That does not make the numbers wrong. It does mean the result should be read as Specific’s claim about Specific’s data, not a neutral finding.
Teams evaluating coding agents for enterprise rollout should treat Real-SWE’s 38.8% ceiling as a directional signal, not a portable number: request the sample tasks, run your own harness against your own codebase, and weigh Fable 5.1’s top score against its $6.96 cost before picking a default.
Reporting and benchmark data from Specific (withspecific.com), authored by Janak Sunil.
