Ramp has assembled a private coding benchmark from 80 tasks drawn directly from its own backend, covering card authorization, bill pay, reimbursements, accounting, procurement, treasury, and fraud. The company built it to test coding agents on the kind of work its engineers already hand off, without the contamination risk that has made public benchmarks less trustworthy as training data absorbs them.
The mechanics separate this from a typical eval. Inspect is the coding agent Ramp runs internally to handle background engineering work; every task in the set traces back to a pull request that agent produced, one an engineer reviewed and merged into the codebase. Building a task starts with pulling the repository back to the commit that existed the moment before that pull request landed. From there, Ramp sets aside two things the agent will never see during evaluation: the code change that solved the problem, and the test edits that proved it worked. The prompt itself comes from a different source than a reader might expect: rather than paraphrasing the pull request’s writeup, Ramp mines what the engineer actually typed to Inspect during the exchange that produced the change, which keeps the ask anchored to what a human really wanted rather than what got written up after the fact.
Scoring follows a strict rule: a submission counts as a win only when the agent’s patch turns every test that was red green, while leaving the rest of the suite intact. Runs that exhaust the context window are counted as failures outright, not excluded or scored separately. That choice matters for a benchmark meant to model background-agent behavior, where an engineer sends a prompt, the agent works unsupervised, and the deliverable is a pull request ready for review on the first pass.
The curation logic is the more transferable idea. Ramp discards any task that every model solves, on the reasoning that a task with no failures carries no signal about which model is better. It treats tasks that no model solves with equal suspicion, but in the opposite direction: rather than assuming the task is simply hard, Ramp’s pipeline flags it as a likely broken test or corrupted environment. The tasks worth keeping sit in the middle of that range, where model performance genuinely diverges and a failed attempt reads as a clean miss rather than infrastructure noise.
Before a task reaches that solve-rate filter, it passes through an audit stage built around a four-way defect taxonomy. Tests can be overconstrained, asserting incidental details like an internal helper name instead of behavior, or they can carry missing assertions that leave real requirements uncovered. Prompts can leak the solution outright, or they can omit context the agent needs to complete the task fairly. Ramp runs the LLM judges that catch these defects on different model providers than the systems being evaluated, a deliberate hedge against a judge favoring outputs that resemble its own family’s style.
The tradeoff Ramp is making sits in plain view. Because every task derives from proprietary production code, Ramp cannot publish the prompts, the gold patches, the tests, or the repository states that make the benchmark work. That opacity is the entire point: it is what keeps the eval out of any model’s training data and prevents the kind of benchmark-specific overfitting that has hollowed out public suites like the original SWE-bench. It is also what makes the benchmark impossible for anyone outside Ramp to audit or reproduce. A reader has to take Ramp’s account of its own pipeline on faith, the same tradeoff every private, production-grounded eval makes by design.
The page Ramp published on August 1, 2026, documents this pipeline in detail: mining merged Inspect pull requests, filtering for implementation-plus-test coverage, validating that gold patches flip tests from failing to passing, synthesizing prompts, auditing with cross-provider LLM judges, and running models from small to frontier before a human makes the final call. What it does not contain is a results table. No per-model pass rates, latency figures, or cost comparisons appear in the material Ramp has made public; the company points to a separate dashboard for aggregate numbers rather than including them here.
Teams building or buying background coding agents should treat Ramp’s methodology as a template worth copying internally, particularly the practice of discarding tasks every model solves and auditing benchmark artifacts with judges from a different provider than the models under test. Whether Ramp’s own agents actually win on this benchmark remains something only Ramp’s dashboard, not this page, can answer.
Ramp described this benchmark methodology on its Labs site, in a page last updated August 1, 2026.
