A spy-game reward teaches AI what no answer key can grade

Reinforcement learning from verifiable rewards, RLVR for short, trains a model by mechanically checking whether its output is correct: a maths result, a passing unit test, a value that matches or it does not. That mechanical check is the whole method, and it is also its limit. Maths and code have answer keys. Summarization, creative writing, and most open-ended reasoning do not, so RLVR stalls at the edge of anything a computer cannot grade by itself.

A paper posted to GitHub by Qinsi Wang and a group of coauthors, accepted to COLM 2026, proposes a way past that wall. Their method, called Reinforcement Learning with Self-Verifiable Rewards (RLSVR), does not build a better grader. It builds a game around the task, one where the rules make the outcome checkable even though the underlying skill is not.

The game is SpyRL, built on the structure of the party game Who Is the Spy. A small group of model instances gets the same assignment: summarize a government report, spin a short story out of a writing prompt, or work out a maths problem tied to a source document. One player, fixed in advance, is secretly the spy and receives a version of the source with a chunk blanked out (a fifth of the text for summarization and fiction, two fifths for maths). Every player produces an answer, then everyone votes on who the spy is. Because the environment, not a person or a preference model, already knows the answer, whether a vote is right is unambiguous. That single fact settles the verification problem: the detection reward is 1 whenever a vote lands on the real spy, normalized across the voting group.

A second reward runs beside it, and it is the sharper design choice. It is zero-sum and built straight from the vote tallies: the spy’s score drops as it pulls more votes than the average civilian gets, and a civilian’s score drops if it pulls more suspicion than peers who worked from identical material. In effect, “write a good summary” becomes “write a summary that reads like it came from someone who saw the whole report,” a standard graded entirely by arithmetic on votes rather than by any model judging quality outright.

The team trained Qwen3-4B-Instruct-2507 and Qwen3-8B with the GRPO algorithm, using the open-source verl framework built by ByteDance’s Seed team, on three corpora pulled straight from Hugging Face with no labeled data: GovReport for summarization, WritingPrompts for fiction, and Nemotron-CC-Math for reasoning. On Qwen3-8B, GPT-4o scored SpyRL’s output ahead of the untrained base model 75.4 percent of the time on summarization and 77.3 percent of the time on creative writing. Two comparable self-play baselines, R-Zero and Absolute Zero, gained only marginally on those same open-ended tasks, per the repository’s own comparison tables. On seven standard reasoning benchmarks, including GSM8K, Math500, AIME 2024 and 2025, Minerva, MMLU-Pro, and GPQA-Diamond, the method lifted Qwen3-4B by an average of 8.97 points and Qwen3-8B by 6.16 points.

That reasoning result is the evidence worth trusting most. None of those seven benchmarks were part of the spy game; the models trained on masked maths documents and were then tested cold on an independent question set with real, checkable answers. That counts as genuine transfer, skill built inside a proxy game showing up on external problems the game never touched.

The summarization and fiction results carry a heavier asterisk. Both are scored as a win rate judged by GPT-4o against the untrained baseline, the same category of judge-dependent signal RLSVR was built to avoid needing during training itself. The authors’ own check on this point, that across 100 games the players who drew more suspicion votes were also ranked lower by GPT-4o’s separate assessment, is a useful correlation but still one model checking another model’s outputs. Whether winning the deduction game reliably means writing better prose, rather than prose that is simply harder to catch, has not yet been tested against anything outside the game.

Today’s issue also carries an argument that verifiability is what let AI crack open maths problems in the first place. This paper is the attempt to manufacture that same verifiability where it does not naturally exist, the identical bet worked from the other direction. Teams evaluating self-play training for domains without an answer key should ask for held-out benchmark transfer, not just a win rate against an untrained baseline, before assuming the game taught the skill they actually wanted.

Qinsi Wang, Jing Shi, Huazheng Wang, Kun Wan, Yiran Wu, Bo Liu, Qingyun Wu, Hai Helen Li, Yiran Chen, Handong Zhao, and coauthors describe RLSVR and SpyRL in “From RLVR to RLSVR,” accepted to COLM 2026 and posted as an arXiv preprint (arXiv