The best AI models still fail nearly four times out of ten at a specific, narrow skill: telling a good next step from a bad one before the outcome is known. That is the finding behind Taste-Bench, a new evaluation released on GitHub by researcher Wenbo Pan and coauthors, alongside a companion paper on arXiv.
Taste-Bench does not test whether an agent can finish a coding or research task. It tests whether the agent can pick the right direction at a fork partway through one, using only what was known at that moment. Each question shows a task, the steps taken so far, and two candidate next moves. Only one of those moves is what a real trajectory later proved correct, whether because a parallel attempt reached a better result or because the agent later doubled back after a dead end.
Researchers mined 4,657 candidate forks and kept 502 of them. Two filters did the pruning: a fork that every model could already answer from the wording alone got cut as too easy, and a fork where the automated judges could not settle on which choice the completed trajectory actually supported got cut as too ambiguous. Of the 502 that survived, 390 come from software-engineering trajectories recorded in SWE-bench and SWE-bench Pro, and 112 come from machine-learning research runs collected through METR’s MALT release, which includes the RE-Bench and HCAST evaluation suites.
The scoring is deliberately hard to game. Each question is asked twice, once in its original A/B order and once with the options reversed and relettered, and a model only gets credit if it answers correctly both times. Guessing randomly nets a score of 25 out of 100. Always picking the same letter position nets a score of zero.
Under the project’s paired_order_v1 protocol, run in August 2026, the leading model, labeled GPT-5.6 Sol on the leaderboard, answered 59.7% of questions correctly. GPT-5.5 was close behind at 59.5%. Claude Opus 5 scored 55.5%, Claude Sonnet 5 scored 51.6%, and Grok 4.20 Reasoning scored 15.7%, a result driven largely by 459 of its 1,004 responses failing to parse into a usable answer at all.
That gap matters more than the headline number. SWE-bench, one of the source datasets here, measures whether an agent can complete a coding task at all, and models have climbed that ladder for two years. Taste-Bench measures something upstream of completion: whether an agent recognizes a wrong branch before it burns through the budget of a long-running task. A model can be fluent at writing code and still be poor at knowing, mid-task, which of two plausible paths is the trap.
The researchers gated the underlying dataset behind a Hugging Face access request specifically to limit training contamination, and they built the scoring pipeline to be reproducible: a command-line tool downloads the data, runs any OpenAI-compatible model against it, and scores the result without needing the original repository’s code.
For teams building autonomous coding or research agents, Taste-Bench is a cheap add to an existing eval suite: a low score here flags an agent that will follow a bad lead for many steps before self-correcting, which shows up in production as wasted compute and slower turnaround rather than an outright failure.
Reported from the Taste-Bench project repository and accompanying paper on GitHub, published by researcher Wenbo Pan and coauthors, August 2026.
