A developer built a 48-cell testing grid to answer a question that has nagged at AI Twitter for two years: are labs quietly optimizing their models to ace one specific joke prompt. Independent developer Dylan Castillo generated 1,008 SVG images across seven frontier models and found no statistically significant evidence that any lab trains toward it. The result matters less for pelicans than for what it says about which informal, crowd-driven benchmarks are still worth trusting.
The prompt in question comes from Simon Willison, the developer and open-source tool builder, who has asked each new model release to draw a bird pedaling a two-wheeler since 2024. Willison posts the results alongside every major launch, and they routinely climb to the top of the Hacker News thread announcing that model, turning an offhand illustration test into an informal public leaderboard.
Castillo, running everything through OpenRouter, widened that single prompt into a grid of eight animals and six vehicles, 48 combinations in all, with the pelican-and-bicycle pairing as just one cell. He generated three samples per combination at temperature 1.0 from GPT-5.6 Terra, Claude Sonnet 5, Gemini 3.5 Flash, Grok 4.5, Qwen3.7-Max, GLM-5.2, and DeepSeek V4 Pro, rendered each SVG to a PNG, and scored the images with a separate LLM judge on animal accuracy, vehicle accuracy, and scene coherence.
His logic: a lab training on the benchmark should show up as pelicans scoring above what their difficulty warrants, bicycles scoring above what their difficulty warrants, or the specific pelican-bicycle cell beating both baselines at once. Castillo fit a fixed-effects regression across all 1,008 images to strip out how hard each animal or vehicle is to draw on its own, then tested every lab’s pelican boost, bicycle boost, and pelican-bicycle interaction against zero. None cleared significance once he adjusted for running 21 separate comparisons. Pelicans actually finished sixth of eight animals on quality, and bicycles finished second-to-last of six vehicles, nearly tied with last-place planes: the opposite ranking a gaming lab would produce.
Castillo’s own caveats keep this from being a full acquittal. A lab could still be improving SVG rendering broadly, which would lift every cell at once, including the pelican-bicycle one, and would look identical to a model that is simply better across the board. Google DeepMind has said publicly that it works on exactly this kind of output quality. A roughly $80 budget also capped the test at three samples per cell and one judge model, wide enough that a modest, deliberate boost could hide inside the margin of error.
Goodhart’s law is the useful lens here: once a measure becomes a target, it stops reliably measuring what it once did. That is why formal benchmarks such as MMLU or SWE-bench now get read skeptically. Labs train against them explicitly, and vendors cite them selectively. Willison’s pelican prompt went viral, got cited in launch threads for two straight years, and still shows no clean sign of having become a training target in Castillo’s data. That durability is itself the signal worth noting: an unofficial benchmark nobody has a financial incentive to game is a rarer and more diagnostic check than a leaderboard every lab already knows to optimize for.
The practical lesson for anyone leaning on public, informal evals to vet a model before adoption is not that all benchmarks are safe from gaming; most formal ones already are not. It is that resistance to gaming should be tested, not assumed, the way Castillo tested it here with a difficulty-adjusted baseline and a real regression. Teams building internal eval suites should apply the same check to whatever screenshot-friendly demo case they rely on before treating it as a proxy for real capability.
Analysis by Dylan Castillo, published July 2026 on his personal blog, dylancastillo.co.
