Independent researcher Ping Lin built three working versions of agent memory, then tested them head to head to see which one actually earns its cost. The comparison matters because most agent products now ship some form of persistent memory, and the architecture choice shapes both accuracy and token spend. The gaps turned out to be large, and one result inverts what the coding-agent industry ships by default.
The first shape is curated files: a small always-loaded index plus topic files the model writes and searches, the approach behind Claude Code’s auto-memory, Cline’s Memory Bank, and similar features in Cursor and Windsurf. The second is a structured store, where every turn gets mined into atomic facts, embedded, and ranked at recall time. That model is what mem0, Letta, and Zep sell as a product. The third replaces the store with trained behavior: a model whose retrieval, judgment, and use of past experience are trained end to end by reinforcement learning. The research system MemHarness is the clearest public example.
Lin ran the file and structured arms behind an identical agent loop, the same open-weight answering model, and the same judge. Only the memory layer could move the score. On LongMemEval, a 500-question suite built to punish weak recall across long histories, the structured store scored 73.6 percent on held-out questions. Files scored 44.9 percent. That is a 28.7-point gap, with a 95 percent confidence interval of 22.1 to 35.4 points. Structured memory also cost less. File-based memory spent roughly 665,000 model tokens per correct answer against 27,000 for the structured store, mostly because curating a markdown file forces the model to reason at write time, while embedding does not.
Files were not worse everywhere. On LongMemEval’s abstention category, questions where the right answer is admitting the history says nothing, files beat the structured store 88.9 percent to 77.8 percent. The same pattern repeated on the second benchmark, LoCoMo. A curation-limited index simply has less material to work with, so the model more often concedes uncertainty instead of guessing. On LoCoMo overall, the verdict flipped depending on whether unanswerable questions counted. Excluding them, structured memory won by 20.5 points. Including them, the gap collapsed to a statistical tie, because ranked retrieval keeps surfacing something plausible enough to tempt a wrong answer.
The trained arm could not join that controlled comparison, since unplugging its experience bank produces a different trained policy rather than a fair baseline. Lin tested it instead on its own ground: two agentic benchmarks called ALFWorld and WebShop. A weak local model and a frontier model, Claude Sonnet 5, each got an experience bank to draw on. The pattern held across both tasks. The weak model gained real, sometimes statistically significant points from retrieved experience. The frontier model, already close to the ceiling, gained nothing detectable. Only the trained policy cleared WebShop’s hardest bar, scoring 87.4 against a range of 63 to 66 for every untrained arm, with or without memory. That score came out of reinforcement learning run against whatever reward the environment itself defined, not from anything retrieval alone could hand it.
Lin flags what the comparison does and does not show. The accuracy gaps rest on paired statistical tests over held-out questions, not on impression. Lin suspects curated files lose even more ground once histories climb well past 500 sessions, but that specific test was too costly to complete. It stays an explicit guess, not a measured result. Swapping the reader-and-judge model moved one benchmark’s score by 6.9 points, more than the gap between two of the three stores, a reminder that these numbers do not transfer across evaluation setups.
This lands the same week AI Insiders covered blast radius over nesting depth in agent hierarchies, and Anthropic’s research on agents failing inside shared environments. Memory is the third piece of that same argument: agents are accumulating state faster than anyone has built discipline for containing it when that state turns out wrong. Teams running a coding agent with a small, human-legible history should stay with files and budget for an abstention check. Anyone building cross-session recall at scale should expect to pay for a structured store. Training memory into weights only pays off if the underlying model is already weak enough to have headroom left to gain.
Ping Lin published this analysis, “The shapes of agent memory,” on their own blog, published in August 2026.
