Google Cloud’s data analytics team argues that the agent benchmarks the industry treats as scoreboards have a structural flaw: once agents start clearing them, the scores stop telling you much of anything. The team, part of Google Data Cloud’s frontier AI group, built a method called Discovery Bench to address that gap, and tested it on the unglamorous but critical job of getting an agent to find the right dataset inside a warehouse full of thousands of tables before it can reason about anything at all.
The core idea borrows a concept from information theory called surprisal: how much uncertainty remains about the correct answer given a specific wording of a question. A test question is only informative if you cannot already predict what the agent will retrieve from it. Google’s illustration is a KramaBench satellite-tracking query that includes the token “TLE.” That single word points almost uniquely to one database table. Remove it and the same question becomes vague enough to match several plausible tables, which quietly destroys its value as a test of judgment.
Discovery Bench generates the same underlying question at three calibrated ambiguity levels, high, medium, and low, instead of shipping one fixed phrasing. The team calls this loop iterative surprisal-based query refinement, or iSQR. Each variant is justified term by term, so difficulty becomes an engineered, measurable quantity rather than a label a human assigned by feel.
The payoff shows up in a retrieval agent built on Gemini 3.1 Pro, run against KramaBench across that ambiguity sweep: an F1 score of 0.34 at high ambiguity, 0.76 at neutral phrasing, 0.81 at medium, and 0.78 at low. One query scored a perfect 1.00 at neutral wording and 0.00 once the word “TLE” was stripped out, with nothing else about the underlying question, agent, or correct answer changed. A conventional benchmark testing only the neutral phrasing would have stamped that case “solved” and reported flat terrain over what was actually a cliff. The sweep also surfaced two concrete weak spots: precision collapsing to roughly 8% when the agent over-retrieved 21 near-identical time-sharded tables for a two-table result, and when a query triggered long retrieval chains, F1 fell from 0.75 to 0.32.
Google’s team then turned the method on itself, twice. First, it audited kramabench-astronomy, a benchmark other teams had already built their own evaluations on top of, and found ground-truth answers that were simply wrong for their queries, at least one question with far more sharded tables than certain retrieval APIs could return, and dates recorded only as months in places that required exact dates. Second, it compared two ways of building the ambiguity sweep itself: one using steering terms an LLM guessed on its own, the other using terms grounded in a TF-IDF measure of surprisal. At high ambiguity, the LLM-guessed sweep scored the agent at F1 of about 0.34; the grounded sweep scored it at about 0.85. Google calls the LLM-guessed version “badly distorted.”
That self-audit is the most useful part of the post, and also where a reader should slow down. Discovery Bench is Google Cloud’s own methodology, tested on Google Cloud’s own agent, using Google’s own model to find flaws in a benchmark other teams relied on. A lab that controls the difficulty dial also controls which score gets published, and the post does not say whether Discovery Bench has been checked by anyone outside Google Data Cloud’s frontier AI team. The team’s own caveat matters here: a difficulty score built from estimated entropy is only as good as the model doing the estimating, which means a team could end up tuning its benchmark instead of its agent.
For any team building or buying an agent product, the actionable move is not to install Discovery Bench specifically but to stop trusting a single-phrasing eval score. Take your hardest real support tickets or search queries, strip the one or two words that make them answerable, and rerun your agent against that harder version. What its authors call “evaluate your evals” is really a warning that a benchmark reporting one pass rate at one fixed difficulty is hiding the cliff, not proving there isn’t one.
Google Cloud published this analysis on its Data Analytics blog, authored by Manav Garg and Sunil Pedapudi of Google Data Cloud’s frontier AI team.
