Vals AI, a company that tests AI models on professional work, has laid out how it will measure whether routers, the software that chooses which AI model answers each request, actually beat using one good model. The results are not out yet. In a post dated 14 October, Vals says a public leaderboard is coming “soon” and invites router makers to ask to be included.
That matters because a router has to earn its place. It adds a moving part between your application and the models, and it only pays off if it is cheaper or better than picking one model and leaving it alone. Until someone runs the comparison on hard work, the savings are a vendor claim.
The one hard number in the post is a ceiling. Vals calculated an “oracle router,” an imaginary picker that knows in hindsight which model was best for every task. Run against the Vals Index, a suite of eight tests spanning tax, law, finance, and coding, that oracle scores 14 points higher than the strongest lone model and spends roughly 40% less. No real product can have that hindsight. The open question is how much of the gap today’s routers close.
The case for routing rests on uneven results at the top. Across 48 models, five hold the leading spot on at least one of the eight tests, and none of them places in the top three on every test. On Harvey’s legal agent test, Meta’s Muse Spark 1.2 comes first despite sitting 33rd overall. Claude Sonnet 5.5 and Claude Opus 5.5 hold places 2 and 3 on the whole Index, yet land 35th and 33rd there. Between the two highest-ranked models, the runner-up runs about 36% more expensive per task. Ten developers also released 16 tracked models in the past month, so any fixed choice goes stale quickly.
On how routers decide, Vals describes one shared design with different brains. Some read the request itself and judge the kind of task or its apparent difficulty. Others lean on outside signals such as benchmark scores or usage statistics. A growing number repeat the decision at each stage of an agent’s job instead of once per task. Ramp’s benchmark routing, which Vals cites, lets a customer choose and weight the benchmarks that steer the pick, so the quality of the choice depends on how closely those benchmarks resemble the customer’s own work.
Per-step routing carries a trap. Switching models mid-session can discard the cached context the previous model had built up, so moving to a lower-priced model can leave the bill unchanged or higher. A weak early pick can also cause errors that take several more calls to fix. Vals argues that academic router tests, which score one choice on one public question, miss this, and that many of those questions are ones frontier models already answer correctly, which makes “always pick the cheapest” look perfect.
The planned method is straightforward. Each router takes the model’s seat inside the same agent setup and is consulted on every call. It runs out of the box, with whatever models and settings it ships with, and Vals will document that configuration. Vals will compare completion time, cache use, cost per task, and task outcome against two baselines drawn from the router’s own pool, the cheapest single model and the strongest one, with the oracle alongside. Six of the Index’s eight tests are private, and practicing lawyers, financial analysts, and tax professionals wrote the tasks and grading rubrics; coding is graded by running tests on what the agent built. The limits are that the tests cover only finance, coding, law, and tax, and that Vals chooses what to publish.
On independence, Vals sells evaluations rather than routing, and it names no stake in any router. The post does not say whether it has commercial ties to any of them, and it is courting router makers as participants. Not Diamond, OpenRouter, and Fireworks have each published their own evaluations, which Vals treats as useful but self-run, and its aim is one shared test run from outside.
Until the leaderboard lands, anyone paying for a router should ask the vendor for cost per task against a single model on their own workload, and treat the 40% oracle figure as a ceiling, not a forecast.
Based on Vals AI’s blog post “Routers Are a New AI Primitive” of 14 October 2026
