Two coding agents each built a working tokenizer trainer in under 30 minutes, and each version cleared every unit test the agent had written for itself. Neither survived contact with the real dataset it was meant to run on. Liquid AI, whose research includes compact language models for edge devices, used that gap between quick success and real failure to test something specific: whether a coding agent can be trusted with a full production build if no human ever reads the code.
Liquid said in an August 18 blog post that the project, a byte-pair encoding tokenizer trainer now open sourced as toktoktok under an Apache 2.0 license, came out of a real need. Its existing options fell short: sentencepiece runs slowly on this style of tokenizer, Hugging Face’s tokenizers library exhausted memory on Liquid’s corpora, and tiktoken cannot train a vocabulary at all. Liquid also wanted two capabilities none of those tools had: restarting from an existing vocabulary and a separate budget per language.
The company picked this task partly because the required skills do not overlap inside its own team. Machine learning researchers there can explain why OpenAI’s cl100k tokenizer sets aside ranks for three-digit numbers, but do not write Rust. Its systems engineers write the multi-threaded, memory-conscious code the trainer needed, but have not trained a tokenizer. Combining the two groups means paying a coordination tax; asking one person to learn the other’s domain first is slow.
Before the agents wrote anything, Liquid set up a short specification document, describing the target and the limits it had to respect rather than how to build it, plus a way to check the result that the agents could not influence. Memory was the stated priority: the trainer had to handle a corpus many times larger than available RAM, treating compute and parallelism as secondary problems. Verification ran outside the agents entirely, on a machine with 128 cores and 2 terabytes of memory, using a harness that checked the finished vocabulary against tiktoken and Hugging Face’s tokenizers, matching token for token across languages, numbers, currency formats, and line endings.
Liquid ran the test with Claude Opus 4.5 and Codex running GPT-5.2, both operating in planning mode with no human review of the code either produced. Both zero-shot attempts loaded cleanly in tiktoken and passed on a small sample corpus. At production scale, each broke differently.
Some failures only appeared once the agents pointed the trainer at Liquid’s real data: certain parquet files store the same column in more than one encoding, and code that read cleanly in testing quietly mishandled that data once it hit scale. Other failures needed a full production run to show up at all, including memory use that collapsed the process near one percent of the target dataset and a whitespace-matching regex that went quadratic on adversarial input.
Three more bugs surfaced only inside the external harness: a merge-rank error that let a vocabulary load without complaint while encoding text incorrectly, a duplicate-merge bug that silently shrank the vocabulary, and a one-character quirk in Rust’s regex engine that broke number handling specifically. That last fix took one line of code. Finding it took the full verification pipeline, not the agent’s own tests.
After more than five iterations with limited progress, Liquid cut the Codex track, still stuck on training throughput, and continued with Claude alone, a call the company frames as resourcing under deadline, not a verdict on either model. Liquid credits Claude’s early version with grasping the reasoning behind the specification’s constraints more consistently than GPT-5.2 did. Claude closed the remaining gaps over several more iterations and produced a trainer that completed the full production job, trillions of tokens across languages and code, in a few days, passing the external harness cleanly.
The detail worth pressing on is why skipping code review was defensible at all here. A trained vocabulary is mechanically checkable: it either encodes and decodes correctly through two independent libraries or it does not, which puts this task closer to a proof checker than to most production software. Most engineering work has no equivalent third party sitting outside the agent’s own code to grade the result. Liquid says the finished repository runs to roughly two thousand lines, all written by its agents and none of it read by a person before publication.
Liquid now treats short, outcome-based specifications and independently verified results as standard practice, extended to open-ended work such as kernel tuning and pull-request triage, where it tracks a metric rather than reviewing code directly. Teams weighing similar autonomy should borrow Liquid’s diagnostic question before trusting an agent alone: whether the task at hand has an outside library, proof, or dataset that can grade the result without a person, or still needs someone to read the diff.
Liquid AI published this account of the toktoktok experiment on its company blog on August 18, 2026.
