OpenAI Says Its Own API Settings Triple ARC-AGI-3 Scores

OpenAI’s GPT-5.6 Sol has cracked unsolved problems in graph theory and finished retro games like Pokemon FireRed, but it stalled at just 7.8% on ARC-AGI-3, a benchmark where AI agents are expected to learn a puzzle game purely by playing it. OpenAI says the model was not the problem. The harness was.

ARC-AGI-3 tests whether an agent can figure out an unfamiliar 2D game through trial and error alone, with no instructions, no tools, and no visibility into how it is being scored. The benchmark’s default harness is deliberately bare, using the same minimal setup for every model tested. On that setup, OpenAI reports GPT-5.6 Sol scored 13.3% on the public task set, far below what OpenAI calculates a typical human tester managed: an estimated 48%, based on ARC’s own published gameplay logs.

OpenAI, in a blog post published this month, traced the shortfall to two specific settings baked into the standard harness. First, the harness deleted the model’s private reasoning after every single move. A reasoning model’s edge comes from the scratch space it uses to plan before acting, and in normal deployment that scratch space carries forward across a session. Stripped of it here, GPT-5.6 Sol had to re-derive each game’s rules from zero on every turn, working only from a bare log of past moves with no record of the thinking that produced them.

Second, the harness capped the running conversation at 175,000 characters and, once that ceiling was crossed, deleted the oldest messages to make room for new ones. OpenAI calls this rolling truncation. In a game that takes dozens of moves to master, the earliest and often most instructive attempts disappeared first, so the model kept relearning patterns it had already worked out. Puzzle-solving of this kind rewards accumulated pattern recognition. A model that keeps losing its own history is handicapped at precisely the task the benchmark claims to measure.

To test the theory, OpenAI reran ARC-AGI-3 through its Responses API with two settings switched on: retained reasoning, which carries a model’s private thinking forward across turns, and compaction, which summarizes older context instead of throwing it away. The public-set score rose from 13.3% to 38.3%, near OpenAI’s estimated human baseline, while output tokens fell sixfold. On the one leaderboard game OpenAI highlighted, no frontier model running the official harness clears the first level. GPT-5.6 Sol did, solving all six.

That result is worth a second look before it becomes marketing copy. ARC built its harness to be generic and free of model-specific tuning, on the reasoning that a plain test keeps comparisons fair and exposes real weaknesses rather than engineering gaps. OpenAI’s finding does not really refute that logic; it steps around it. The company is explaining away a weak score on somebody else’s public leaderboard by rebuilding the test around its own infrastructure, then recommending that future evaluators adopt those same settings. That advice can be technically sound and commercially convenient at once, since a benchmark tuned to OpenAI’s preferred API configuration is a benchmark that plays to OpenAI’s product design.

Some credit is due on the other side too. OpenAI did not simply announce a higher number. It named the two mechanisms, published a documented before-and-after comparison, and credited ARC’s own analysis for prompting the investigation rather than claiming an unprompted discovery. The harder question, whether a harness that erases a reasoning model’s memory measures the harness more than the model, is one ARC and every rival lab chasing agentic benchmarks will eventually have to answer, regardless of who raised it first.

For anyone citing a benchmark score to justify a model choice or a budget line, the practical move is to ask what harness produced the number before trusting the headline. A 13.3% score and a 38.3% score describe the identical model, separated only by configuration, not capability.

OpenAI researchers Ilan Bigio and Ted Sanders detailed these ARC-AGI-3 findings in a company blog post published in July 2026.