A new benchmark called dig.bench does not ask a model to answer a question. It drops the model into a text-based game whose rules are never explained, gives it a fixed number of steps, and checks whether it wins. Most evaluations measure recall: what a model already knows from training. This one measures something rarer: whether a model can experiment its way to knowledge it does not have.
The project comes from a research group that includes Tri Dao, Jürgen Schmidhuber and Joshua Tenenbaum, according to the citation posted on the benchmark’s own site. Their paper, “DiG-bench: Discovery in Games,” frames the work as a test of scientific discovery: an agent must probe an unfamiliar system, form hypotheses about how it behaves, and apply what it learns before the step budget runs out.
The design choices are deliberate. Every game is text-only, which the authors say removes visual confounds so that discovery, not perception, is the only variable being tested. Humans and models play through an identical interface: the same game states, the same available actions, the same step limits. Some games include a creative mode, a separate level where a player can experiment freely without those trials counting against the step budget, before attempting the timed run.
The benchmark holds most of its material back. It contains 70 games organized into seven difficulty tiers of ten games each, but only 21 are public. Withholding 49 games is a contamination defense: a model that has memorized solutions from a public leaderboard cannot lean on that memory for the majority of the test. For a benchmark whose entire premise is that the model has never seen the rules before, that withholding is not incidental. It is the mechanism that keeps the test honest as labs train on ever more of the public internet.
The result the benchmark is built to surface is the gap at the top. All 70 games were validated as beatable by external human testers on their first attempt, across every tier, including the hardest. A first-time human tester can still close out the hardest tier within the step limit. Frontier models, working through the same interface with the same budget, mostly fail to close those same games out. That gap, not any single leaderboard number, is the finding worth watching. The public leaderboard lists two harness types, a basic harness running Opus 5, GPT-5.5, Kimi K3, Gemini 3.1 Pro, GLM-5.2 and others directly, and an agentic harness pairing models like Opus 5 with Prime Agent or GPT-5.6 Sol with Codex. The site does not publish per-model win rates in the material reviewed here, so the size of the human-model gap on the hardest tier cannot yet be independently checked outside dig.bench’s own reporting.
That gap matters beyond puzzle games. Discovering unstated rules under a step budget is a reasonable proxy for what an exploratory agent does in the real world: probing an undocumented API, debugging a system with no manual, or navigating a codebase whose behavior only reveals itself through trial and error. A model that reliably solves dig.bench’s easy tiers but stalls on the hardest ones is a model that can execute known playbooks but struggles when the playbook does not exist yet. That is a narrower skill than fluent conversation or benchmark recall, and it is a skill agentic products increasingly depend on.
Teams evaluating an agent for genuinely novel environments, rather than tasks with documented APIs and prior examples, should treat top-tier performance here as more diagnostic than aggregate scores on knowledge benchmarks, and should watch for whether labs start reporting their own dig.bench numbers once more of the 49 held-back games see wider testing.
According to dig.bench, the benchmark’s own site, where the leaderboard and methodology described here were published.
