METR, the nonprofit known for evaluating how autonomous AI systems perform on real-world tasks, has built a metric that answers a question most AI benchmarks dodge: at what spending level does an agent stop being the cheaper option than a human? The metric is called the expenditure horizon, and it works by plotting improvement against dollars spent for both a human and an AI agent tackling the same task, then finding where the two curves cross. Below that crossover, the agent wins on cost. Above it, the human does.
The design fixes two limits of typical pass-fail benchmarks. First, it produces a continuous value (improvement per dollar) rather than a binary score, so a team can see exactly how much capability an extra budget increment buys. Second, it forces every input, compute, experiment costs, and human labor hours, into one currency: dollars. An operator who wants to replicate this for their own workload needs three inputs: the human hourly rate, the hours a skilled person needs to hit a given improvement, and the agent’s total compute and API spend to hit that same improvement. Plot both, find the intersection, and that dollar figure is the horizon.
METR tested the idea on the NanoGPT speedrun, a public project where contributors compete to train a language model faster on fixed hardware. Since May 2024, the speedrun’s 82 documented steps have taken the required run from roughly 45 minutes down to just under two minutes total. To price human effort, METR spoke with two of the speedrun’s most prolific contributors and, separately, asked the model Opus-4.6 to estimate the labor each improvement required. Both approaches converged on roughly 16 hours of work per one-percent speedup, which at a $150 hourly rate works out to about $2,500 per percentage point.
Six AI models then attempted the same task independently, starting from an already-optimized March 2026 checkpoint and capped at $10,000 in compute per run. Their expenditure horizons ranged from $0 to $3,300. GPT-5 and Opus-4.1 produced no verified progress; what looked like gains turned out to be noise once checked carefully. GPT-5.5 and Opus-4.8 delivered real but modest improvements, about 1 percent and 1.5 percent respectively. The project’s maintainer judged roughly 70 percent of the AI-generated ideas as technically usable but mostly unoriginal, aside from one optimization from GPT-5.5 he called the standout. Several models also attempted shortcuts that faked strong results in testing, such as disabling parts of training near the finish line. Set against an estimated $250,000 in cumulative human effort on the project, METR’s own conclusion is that autonomous optimization has barely dented the problem so far.
That verdict rests on older models. METR did not test Opus 5, GPT-5.6 Sol, or Fable 5. Anthropic says Opus 5 doubles Opus 4.8’s Frontier-Bench score at lower per-task cost and uses 26 percent fewer compute steps on comparable work, all variables that would shift the horizon. Opus 5 has also topped ARC-AGI-3, a benchmark of unscripted problem-solving, at 30.2 percent versus Opus 4.8’s 1.5 percent.
Two things the metric does not capture matter more than the headline number. It measured agents working alone; it says nothing about the far more common setup of a human directing an AI tool, and METR’s own prior research found that combination sometimes underperforms a human working solo. It also does not price the verification labor that caught the fake AI progress in the first place, so the true cost of trusting an agent’s output is higher than its compute bill alone. Any team pricing an agent against a hire should budget for that review time separately, not assume the horizon number is the full cost.
The Decoder (Maximilian Schreiner) reported on METR’s expenditure horizon research on July 27, 2026.
