Picking a coding-agent harness once and keeping it while you change models can quietly inflate your bill. Runta, the cloud runtime company, published FrontierHarness Eval v1.1 on November 17, and its main finding is that the cheapest harness for one model is often a poor choice for another.
A harness is the software wrapped around a model that lets it do real work. It supplies the system prompt, the tools for reading and editing files and running shell commands, and the loop that feeds each result back to the model until the task ends. Codex CLI, OpenCode, and Pi are all harnesses. The model supplies the intelligence; the harness decides how that intelligence gets spent.
Runta built the study with Baseten, which served all five models. The test covered 22 harness configurations (18 harnesses, with OpenCode run at two versions and four DeepSeek Harness profiles counted separately), five open models, 62 terminal tasks, and four attempts each. That adds up to 27,280 agent runs and 83 billion tokens. Runta says this is 76 times the size of its v1.0 leaderboard, which ranked nine harnesses on a single model, Kimi K3.
The ranking reshuffled badly from one model to the next. Codex CLI was the cheapest per task with GLM-5.3 and landed 16th of 22 with Kimi-K3. DeepSeek Harness: Creator fell from first with Muse to 21st with GLM Flash. The median harness moved 11 places when the model changed. Runta also split the attempts into two halves to rule out noise: rankings under the same model correlated at 0.71, but across two different models only at 0.26.
Runta’s best-value pick, which weighs pass rate against cost, therefore differs by model: Codex CLI with GLM-5.3, Pi with GLM-5.3 Flash, ZCode with DeepSeek V4.1 Flash 1017, Oh My Pi with Muse, and Alma with Kimi-K3. Confidence varies. Codex CLI kept its GLM-5.3 win in 99.6 percent of 4,000 resamples of the task set, while Alma’s Kimi-K3 win held in only 25 percent.
The model still dominates the economics. An analysis of variance on the cost of 110 harness and model pairs attributed 92.8 percent of the spread to the model, since list prices differ by 20 times or more. The harness on its own explained 2.3 percent, and the specific pairing plus noise explained 4.9 percent. Yet once the model is fixed, the harness matters a lot: the priciest harness cost 1.8 to 5.0 times the cheapest per task. The widest gap came with DeepSeek, where Cline CLI cost 5.0 times as much as OpenCode v1 for 2.0 more points of pass rate. Runta shuffled trials between harnesses 300 times to test whether chance could produce such gaps. Chance gave 1.2 to 1.8 times, and all five real ratios exceeded every shuffle (p = 0.003).
Spending more did not buy better results. Across the models, the rank correlation between cost and pass rate ran from -0.39 to +0.22. The harness with the top pass rate ranked anywhere from 5th to 20th of 22 on cost. Across all models, ZCode posted the highest pass rate at 61.9 percent and OpenCode v2 the lowest cost per task at $0.561.
A leaner harness is not automatically a cheaper one. The length of the prompt, the number of tools, and how often the cache was hit showed no link to cost per pass (correlations of -0.03 to +0.02). Steps did: the correlation was +0.51 (p = 0.015), because every step resends the full context. The smallest opening request, 882 tokens, belongs to mini-SWE-agent. It still needed 38 steps per pass and ranked 17th of 22 on cost per pass.
Failure is where the money goes. Runta found that 64.5 percent of list-price cost went to trials that did not pass (52.7 percent to failures and 11.8 percent to timeouts). On the five harnesses Runta rated best value, 68 percent to 82 percent of failed runs closed by declaring an answer instead of running out the clock, and the verifier then reported a failure. Runta did not test a pre-exit check, so any saving there is unproven.
The limits are real. All five models are open, served by one provider, and Runta’s costs are estimates at Baseten’s list prices. Each pass rate carries a 95 percent interval of about plus or minus 10 points, so small gaps are ties. Runta plans to add Claude Opus 5.5, GPT-6 Astra, and Gemini 4 Carbon.
Any team that moves a coding agent to a different model this quarter should rerun its own tasks on two or three harnesses before signing off, because a 1.8 to 5.0 times cost spread is easy to leave on the table.
Source: Runta’s blog post “Introducing FrontierHarness Eval v1.1,” published November 17, 2026.