Running the same model through a different piece of orchestration software can cost twice as much per task without making it meaningfully better at the work. That is the headline finding from HarnessTax, a study Arena published on its blog on September 16, credited to Melissa Z. Pan, S. Yang, N. Arabzadeh, W.-L. Chiang, Ion Stoica and Matei Zaharia.

The researchers tested 21 combinations of seven models and three harnesses: Claude Code, Codex CLI and Pi. Each pairing ran the same 30 sampled tasks from SWE-bench Lite and Terminal-Bench 2.0, three attempts per task, every harness set to its high-effort mode with a ceiling of 100 agent turns per run. The price tag for each attempt came from a fixed direct-API rate card dated September 1.

Anthropic’s Claude Fable 5 illustrates the pattern. It closed 97.8% of attempts in Claude Code, effectively tied with Codex and Pi, both of which landed at 96.7%. The price did not track that near-tie: Claude Code averaged $1.33 per attempt against $0.67 in Pi, almost exactly double for a fraction more success. Averaged over every model the two benchmarks had in common, Claude Code ran about 2.0 times the cost of Pi and 1.6 times Codex on SWE-bench Lite, and 1.5 times Pi’s cost on Terminal-Bench 2.0. The paper’s authors call that gap a “harness tax”: harness choice moved success by no more than 2 percentage points on SWE-bench Lite and roughly 5 points on Terminal-Bench 2.0, far smaller than its effect on the bill.

Pi is the harness doing the least, by design. Its toolkit stops at reading, writing, editing and running commands in bash, and the study still places it on the frontier of cost versus success on both benchmarks. Turn counts help explain why the price gap opens without a matching capability gap. Fable 5 needed almost the same number of turns in either harness, 15.4 in Pi against 15.3 in Claude Code. What differed was the starting overhead: averaged over the full set of seven models, Claude Code entered each task carrying more than ten times the context load that Pi used, the product of longer instructions and heavier tool schemas billed on every single turn regardless of outcome.

A third finding undercuts the assumption that a lab’s own model performs best inside that lab’s own agent. Look at the six models built by Anthropic and OpenAI across both benchmarks, twelve comparisons in total, and a rival harness beat the home product in nine of them. Sonnet 4.6 cleared 68.9% of SWE-bench Lite tasks running in Codex against 66.7% for the same model inside Claude Code, at comparable cost. Pi got more out of GPT-5.6 Sol too: 83.3% on Terminal-Bench 2.0 versus 78.9% for that model in Codex, and at roughly half the price ($0.42 against $0.76). GPT-5.6 Luna came in cheapest on both benchmarks. The open-weight Kimi K3 stayed close behind it, holding a similar position near that cost-success frontier.

Arena sponsored the API access behind the experiments, and the authors are explicit about the limits of what they measured. Thirty tasks per benchmark is a small sample, both benchmarks are open source and may already sit inside the models’ training data, and a heavier harness could still earn its keep on workloads these two benchmarks do not represent. The team says it plans to release the underlying profiling traces and argues that harness selection should eventually be handled automatically rather than left to a developer’s default choice.

For any team paying per token for agentic coding at scale, the practical move is to benchmark the cheapest harness against whatever ships as the default first. This data suggests that default is priced for capacity most teams are not using.

Arena published the HarnessTax study on its blog on September 16, 2026.