Microsoft finds coding agents do worse with JSON than plain CLI args

Microsoft ran a controlled test to settle an argument circulating among developer tooling teams: should a command line tool built for AI agents drop its individual flags in favor of a single JSON payload. The company’s developer blog published the results on July 7, and the answer was no, consistently, across every model tested. That undercuts a piece of common advice telling CLI maintainers to restructure their tools around JSON because agents supposedly reason more naturally in structured formats.

To test the claim, Microsoft built a synthetic CLI called podctl that creates multi-service deployments, a deliberately messy scenario with three levels of nesting, arrays, and more than 30 distinct values. Two separate binaries ran on the same validation backend: one accepting only individual flags, one accepting only a JSON payload, both renamed to podctl so the agent could not tell which version it faced. Five coding models ran the task five times each through GitHub Copilot Chat: Claude Haiku 4.5, Claude Sonnet 4.6, Claude Sonnet 5, GPT-5.3-Codex, and MAI-Code-1-Flash.

Flags won on correctness outright. Every model, including the smallest, Claude Haiku 4.5, produced a correct deployment in all five runs when using args. JSON told a different story. The three strongest models, Claude Sonnet 5, Claude Sonnet 4.6, and GPT-5.3-Codex, matched that perfect record, but MAI-Code-1-Flash dropped to three correct runs out of five, and Claude Haiku 4.5 managed only two.

JSON cost more in every single case. Per task, GPT-5.3-Codex went from $0.05 with args to $0.54 with JSON, an 11 times increase. Claude Sonnet 5 showed the smallest gap, four times, moving from $0.08 to $0.32. Across all five models the JSON path cost 4 to 11 times more, driven mainly by output tokens: when a model’s JSON gets rejected, it has to reason about the error and try again, and Microsoft found those retries generated 7 to 14 times more output tokens than the args path.

Retries traced back to a specific mechanical problem: shell escaping. Microsoft’s primary run used Windows with PowerShell as the default agent shell, and Claude Haiku 4.5 illustrated the failure mode clearly. Three of its five JSON runs never produced a working invocation at all. The model wrote valid JSON, then could not get quoted strings past the shell, cycling through base64 encoding, temp files, here-strings, and cmd.exe passthrough before giving up.

Shell choice mattered more than model choice in some cases. Microsoft reran Claude Sonnet 4.6 on macOS, where agents default to Bash, and the cost gap between args and JSON collapsed from 9 times to 1.5 times with the same model and the same payload. Flags stayed flat regardless of shell: $0.05 on PowerShell, $0.07 on Bash. JSON’s cost premium is partly a Windows tax, meaning any team benchmarking agent workflows on one operating system is measuring only half the problem.

The structural explanation matters most for tool builders. A flag interface hands the agent a constrained menu: the help text names every valid option and its type, so the model only maps task language onto a known set of names. A JSON interface hands the agent a blank canvas that must satisfy a schema through correct syntax, correct nesting, correct field names, and surviving the shell. That is four separate places to fail instead of one. Narrowing the input space compensates for gaps in model capability; removing that constraint does not help strong models and actively breaks weaker ones.

The practical guidance for anyone shipping a CLI, SDK wrapper, or MCP tool that agents will call: keep flags as the primary interface, since they held up across every model tier and both shells tested. A JSON option can coexist for programmatic or batch use, but deleting args to chase an agent-friendliness assumption is not supported by the data, which showed JSON never improved correctness and always raised cost. Anyone weighing that rewrite should run a same-day comparison across their own target models and shells before touching an interface that already works.

Microsoft’s developer blog, published July 7, 2026, in a post by Principal Developer Advocate Waldek Mastykarz.