An agent that passes a task once does not reliably pass it again, according to a technical blog post from IBM Research published on Hugging Face. The company says a ReAct-style agent running on GPT-4.1 solved 77.4 percent of tasks on average across five repeated runs on the AppWorld benchmark, but succeeded on every single one of those five runs for only 53.0 percent of tasks. IBM Research calls that 24.4-point spread the “consistency gap,” and says it widens to roughly 30 points on the hardest task tier.
The distinction matters because most agent benchmarks report only the average, a metric IBM Research’s post labels Mean@k. That number answers how good an agent is over many attempts, not whether it will repeat a specific success the next time a user asks the same question. The post introduces a second metric, Pass^k, defined as the share of tasks an agent completes correctly on every one of k runs. By that stricter measure, an agent can look strong on a leaderboard while still failing the same request roughly one time in four.
IBM Research attributes the flips to how sharply an agent’s underlying token probabilities favor one action over another at each decision step. When two options are nearly tied, the company says small platform-level variations, such as floating-point rounding differences or how requests get batched on a server, are enough to tip the model toward a different path. Because a single task can chain dozens of such decisions, IBM Research argues the odds of at least one flip compound quickly, even when the agent runs at temperature zero, the setting typically used to suppress randomness.
To address the gap, IBM Research built a tool it calls the Consistency Analyzer, which is now packaged as a new “consistency guidelines” feature inside its open-source ALTK-Evolve framework. Rather than rerunning a task end to end, the analyzer replays a single recorded trajectory and resamples each decision point, requesting five completions per step by default to spot where the model was close to choosing differently. Those flagged points are converted into written guidelines that get fed back to the agent at inference time, the same mechanism ALTK-Evolve already used to improve raw task success in an earlier release IBM Research cited.
The company reports that on AppWorld’s 168-task test set, the guidelines cut the consistency gap roughly in half, from 24.4 to 12.0 percentage points, while also raising average accuracy from 77.4 to 81.0 percent. Medium-difficulty tasks gained the most in absolute terms, and IBM Research says the guidelines transferred to related tasks outside the ones they were generated from, including a smaller open-weight model, gpt-oss-120b, where similar-task gains exceeded same-task gains. These are IBM Research’s own benchmark results, run on its own methodology, and have not been independently replicated.
For teams running agents against repeatable workflows, such as processing the same class of financial or contractual request over and over, a benchmark’s headline accuracy number says little about how often that exact workflow will fail on a second try. Operators evaluating agent frameworks should ask vendors for Pass^k alongside Mean@k, and treat any agent shipped without that number as unverified for production repeatability.
Based on a technical blog post by IBM Research, published on Hugging Face on September 16, 2026.
