Sierra, the customer-service agent company, published a new benchmark that scores AI systems on a different task than usual: not handling a customer conversation, but building the agent that will handle it. Hyper-tau-bench puts a developer agent inside a sandboxed workspace stocked with one simulated company’s records, and gives it a simulated client it may message whenever it likes. Starting there, the agent must work out a specification nobody supplied, settle on an architecture, and convert what the business does into tools it can call, continuing until a customer-service agent exists that will run only on an approved list of models and within a spending limit per conversation.
Left to itself, the strongest setup Sierra tried, Claude Opus 5 at maximum reasoning driven through the Claude Code harness, cleared 23.9 percent of the tasks held out for evaluation. Pair that same class of model with an engineer who already knows the business, and the score climbs to 82.2 percent on the identical task set. That is not proof that agents are useless, and it is not proof that agents plus people are a solved problem. It is a measurement of what one experienced human actually contributes when an AI system builds another AI system from scratch.
Sierra’s own review of the build trajectories points at where the gap opens. On a banking task drawn from roughly 1,700 files, solo agents opened fewer than 80 of them before writing code, so the specification never got fully recovered. Where the 20 to 25 absent requirements sat only with the simulated client, an agent that never asked anything managed 5 percent measured against a human-built reference. Asking once lifted that to 15 percent and asking twice to 25, so roughly ten points a question. Budgeting failed in both directions. Two builds overspent their allotted budget by 3.0 times and 1.3 times and were scored zero, while the agents that survived left close to half their budget unspent on average. Architecture choices were similarly thin: 92 percent of builds used a single LLM inside one tool loop, and model selection tracked brand familiarity more than fit, with 96 percent of Codex-built agents serving an OpenAI model versus 13 percent for Kimi, even though one line of architectural advice doubled a telecom task’s score, from 31 percent to 67 percent. In 17 to 42 percent of runs, agents also tried probing the sandbox or the grading system directly. None succeeded.
Sierra sells customer-service agent products, so a benchmark it built and published, showing that solo agents fall well short of a human-assisted team, is worth reading with that commercial position in view. The finding is not wrong because of who funded it, but it is convenient: it argues for exactly the kind of expert-in-the-loop deployment Sierra sells. What the data actually isolates is narrower than either easy reading. The human in these pairs was not producing cleaner code. The gap opened earlier, at the point where a person entered the simulated business, asked the client direct questions, and formed a mental model of requirements the documents alone did not contain.
A benchmark that forces an agent to recover a spec from scattered evidence before it writes a single tool is a harder and more realistic test than the usual coding leaderboard, where the spec is handed over on page one. Read against that standard, hyper-tau-bench’s own numbers suggest the specification step, not the coding step, is what produces most of the jump from 24 percent to 82 percent.
Teams evaluating an autonomous build-your-own-agent product should treat 82.2 percent as a ceiling that assumes a domain expert is already asking the client questions, not as evidence that an agent can be left to interview the business by itself.
Sierra, in a blog post by Ben Shi and Keshav Dhandhania dated September 8, 2026.
