Jason Liu argued on X on July 12 that GPT-5.6 Sol is underrated for work outside of coding, pointing to long-running tasks across apps, browsers, and enterprise data as the model’s real strength. The claim cuts against the dominant way AI capability gets measured in public, which is coding benchmarks, because those are cheap to score and easy to summarize in a headline. If Liu is right, the industry’s leaderboard obsession may be measuring the wrong slice of the job.
Coding benchmarks exist because coding has a clean pass or fail signal: a test suite runs, or it does not. Long-horizon knowledge work, the kind Liu describes, such as configuring a workflow inside an enterprise app, supervising a multi-step process, or reasoning across an extended browser session, has no equivalent scorecard. That absence does not mean the capability is not real. It means nobody has built the instrument to measure it, and the labs racing to publish new benchmarks each month have little reason to fund one that would slow their own launch cycle.
This is a real structural gap, not a minor one. Software engineering tasks are overrepresented in public evals partly because they are easy to grade automatically, which means models get tuned toward what is measurable. A model that quietly handles a week-long enterprise workflow without dropping context could be genuinely undervalued by a discourse built around SWE-bench scores and coding leaderboards. Liu’s underlying observation deserves to be taken seriously on those terms.
The specific evidence Liu offers is weaker than the observation. He says internal teams used GPT-5.6 Sol to configure and supervise the training of a system he calls Luna. That is a claim about how a team used a tool internally, not an independent evaluation with a control group or a published task set. It reads closer to a testimonial than a benchmark, and testimonials from people close to a product’s success are among the least surprising endorsements in AI right now.
Liu also points to Ultra mode, which he says adds sub-agents that parallelize pieces of a complex task for faster, stronger results. That architecture detail is worth examining on its own terms, separate from who is praising it. Sub-agent orchestration is the direction most frontier labs are already moving, because a single long context window still degrades in quality the longer a task runs, and splitting work across coordinated sub-agents is one of the few approaches that has shown consistent gains on multi-step problems.
None of this is evidence to import wholesale. A post from one practitioner, however well-informed, is not a substitute for a benchmark, and “we used the model to supervise our own model’s training” is a claim about internal usage that nobody outside the company can check. It should be read as a hypothesis worth testing, not a verdict already reached.
The useful move for an operator is neither hype nor dismissal: build a long-horizon task evaluation that scores a real multi-step workflow spanning apps, a browser, and internal data, then run GPT-5.6 Sol against whatever model currently sits in your stack. If it holds up under your own numbers, you will have found signal that the coding leaderboards were never designed to surface.
Based on a post by Jason Liu on X, July 12, 2026.
