An eleven-person research team released PACE, a framework that predicts a model’s score on expensive agentic benchmarks by testing it on a small, curated subset of cheap, non-agentic tasks instead of running the full benchmark suite. The paper, posted July 2 and later surfaced on Hugging Face Papers, targets a cost problem that has quietly shaped how labs evaluate models: agentic benchmarks like SWE-Bench (a coding benchmark) and GAIA (a general-assistant benchmark) can cost thousands of dollars and take days per run. PACE claims to reproduce those scores at much less than 1% of that cost, with a mean absolute error under 4%.
The mechanism is a fitted regression, not a new benchmark suite. PACE draws candidate items from 19 existing non-agentic evaluations, ones that test isolated capabilities such as reasoning or code generation, and selects the specific instances whose scores correlate most reliably with a target agentic benchmark’s outcome. Two selection strategies feed the pool: one that targets instances closely related to the agentic task, and one that favors instances that carry broadly useful signal across many benchmarks. The resulting compact test set is what the authors call PACE-Bench.
The reported numbers come from testing across 14 models, four agentic benchmarks, and 19 non-agentic source benchmarks, using leave-one-out cross-validation. Beyond the sub-4% mean absolute error, the authors report a Spearman correlation above 0.80 between predicted and actual agentic scores, and roughly 85% accuracy when the proxy is used only to rank one model against another. That ranking number matters more than the raw score prediction for most practical use: teams choosing between two model candidates usually care which one wins, not the exact percentage gap.
This changes the arithmetic behind a decision every AI team already makes. Full agentic evaluation has functioned as a gate that only well-funded labs and enterprises could afford to run repeatedly, since testing every checkpoint against SWE-Bench and GAIA in full is not something a smaller team can do on each release. A proxy that costs a rounding error of the original price opens up per-checkpoint agentic scoring, live model routing decisions, and vendor comparison shopping to teams that previously relied on published leaderboard numbers they could not verify themselves.
The paper’s own framing invites a specific caveat. PACE’s regression is fitted and validated on 14 models that already existed when the researchers ran their experiments. A proxy of this kind works by learning which cheap-benchmark signals move in step with expensive-benchmark outcomes for the model families it has seen. The paper does not establish whether that correlation holds when a frontier lab ships an architecture meaningfully different from the ones in the training set, rather than a scaled variant of a familiar design. If the underlying skills that separate models on SWE-Bench and GAIA shift with a new architecture class, the regression’s coefficients would need to be refit against fresh full-benchmark runs before anyone could trust the proxy again, which reintroduces the very cost the tool exists to avoid.
A second drift risk sits outside model architecture entirely. The four agentic benchmarks PACE was proxying will themselves get updated versions over time, and each update would need a corresponding recalibration of the proxy’s instance selection, a cost the paper does not quantify.
Teams currently paying full price to run SWE-Bench and GAIA before every release should pilot PACE-Bench on their next evaluation cycle, but should keep periodic full-suite runs in the loop specifically to confirm the proxy’s correlation is holding for their own model lineage before retiring the expensive benchmark entirely.
Based on “PACE: A Proxy for Agentic Capability Evaluation” by Yueqi Song and colleagues, posted to arXiv and Hugging Face Papers on July 2, 2026.
