Stripe has released a benchmark that scores how well AI coding agents build Stripe integrations, and the headline finding is a gap between looking right and being right. According to a post on Stripe’s developer blog, agents averaged 78.1 percent on partial credit but only 29.4 percent when every required criterion had to pass.

The test set holds 69 cases covering single charges, recurring billing, and Connect, its marketplace and platform product. Each scenario was tested on 14 pairings of model and harness, spanning Claude Code, Codex, and OpenCode, with two runs each, with and without the Stripe agent plugin. The plugin bundles Stripe-written skills with an MCP server that offers an implementation planner and documentation search.

Stripe’s stated motive is that agents lean on older APIs because those dominate their training data, and that ordinary coding benchmarks cannot see the problem. A test suite can go green while the integration uses a deprecated pattern. Stripe’s example: GPT-6.1 Sol, working on a mobile physical-goods scenario without the plugin, reached for a recently removed card-only parameter. With the plugin it used automatic payment methods instead. Across all runs, Stripe says outdated parameter references fell from 7 percent to 1.3 percent of parameters in API calls.

The plugin helped everywhere. All 14 pairings gained from it, between about 3 and 8 points. Web searches dropped 96 percent, which suggests agents stop hunting the open web once a trusted source sits in the tool list. Behavior diverged by model, though. Fable 5.1 made 53 percent more tool calls with the plugin, while Gemini 3.8 Flash made 26 percent fewer.

The strict numbers are the more useful read for anyone shipping money-moving code. The best pairing with the plugin, GPT-6 Astra in Codex, passed every required criterion in 52.9 percent of cases. So even the top result fails nearly half the time. Stripe’s own diagnosis is that agents produce almost-correct work and stumble on the last step. One example: in a fulfillment scenario, Opus 5.5 checked only that a payment had succeeded, not whether the order was still eligible to ship, which could release goods after a refund or dispute.

Results also split by product area. Agents that led on one-time payments and subscriptions did not necessarily lead on Connect, and the Connect scenarios, which demand more integration decisions, gained the most from the plugin.

Three caveats sit in Stripe’s own methodology. First, the benchmark checks specification, not execution: a well-shaped integration that never runs can still score full marks. Second, a role-playing agent standing in for the developer, and the judges grading each criterion, both ran on GPT-6.1 Sol, one of the models under test. Stripe says a probe using Opus as judge showed no bias, only a more lenient scorer. Third, the plugin is Stripe’s product and the benchmark is Stripe’s design, so that lift is a vendor measurement. Stripe is publishing the benchmark and its Forge eval harness, which lets outsiders rerun it, and says it will keep adding models and cases.

The practical reading is a scoreboard for the agent stack as much as for Stripe. A team that lets an agent write its billing code today is accepting something like a coin flip on strict correctness, plugin or not. Before merging agent-written payment logic, run the refund, dispute, and webhook paths by hand, and treat any partial-credit score as a ceiling, not a forecast.

Stripe, “Stripe evals: Measuring real-world coding agent performance,” stripe.dev blog, retrieved November 20, 2026 (the post carries no publication date).