Microsoft published AutoSaddler, an open source system that automatically rewrites an AI agent’s prompts, tool definitions, and middleware based on its own failed execution traces. The target is the scaffolding around a model, not the model itself, and that scaffolding is where most production agent failures actually originate. Two teams running the identical model on the identical task routinely get different results because of how tools are described, how errors get surfaced back to the model, and how much context the agent loop retains.
AutoSaddler runs three session types in sequence. A diagnosis session inspects failed traces alongside the harness codebase and proposes either patches to code and infrastructure or patches confined to prompt wording and instructions. A reflection session then compares traces from before and after a patch, bucketing every case as repaired, broken by the change, failing as it did before, or passing as it did before. A third session draws on the accumulated history of prior patches, a lineage graph the project calls an EvoDAG, to combine working components rather than starting each fix from a blank page.
Every candidate fix is checked against a sample of training cases and then gated on a held-out development split before it is kept. The repository reports preliminary test scores on three benchmarks: GAIA2 pass rate moved from 53.0 to 62.0 on a default ReAct agent, SWE-Bench Pro moved from 37.3 to 46.9 on SWE-agent, and Terminal-Bench 2.0 moved from 40.0 to 50.0 on Terminus 2, each roughly a nine to ten point gain. These numbers come from Microsoft’s own runs against its own benchmark splits. The README does not cite an independent replication.
The failure taxonomy is the more useful signal than the score gains. AutoSaddler assumes agent breakdowns split cleanly into two categories: a tool description that omits an edge case or a missing retry path, which needs a code fix, versus a prompt that under-specifies when to stop, which needs a wording fix. The project applies those two categories in a fixed order it calls Capability-to-Steering, patching structure before patching language. That ordering is itself an argument about where agent failures start: not in the model’s reasoning but in the interface between reasoning and the tools it calls.
The current release, versioned V2, is built for durability rather than one-off experiments. It logs an append-only event stream, keeps immutable provenance for every candidate, and can resume or fork a run from a validated checkpoint. Configuration is strict by design: reusing a run ID is allowed only when every resolved input, every source revision and every setting matches the original byte for byte, and any mismatch is rejected outright rather than silently overwritten. Supported integrations today are narrow: a deterministic test harness plus a Meta-ARE ReAct agent evaluated on GAIA2, with Terminal-Bench and additional harnesses listed as coming.
This is a concrete instance of a broader shift: as frontier labs ship roughly comparable model weights to every customer, the gap between a working agent and a broken one now sits mostly in the harness around it, not in which model runs underneath. Teams building or buying agent products should treat prompt and tool-schema quality as a measured variable, not an afterthought, and should start logging execution traces now, since a diagnosis pipeline like this one is only as good as the failure data it has to work from.
Published by Microsoft on GitHub, 25 August 2026.
