Google Research has published a method that lets a language model keep learning after it ships, by alternating between working and a rest phase the team calls sleep. Today’s models are frozen on release day, so anything new has to be pasted into the prompt every time. The authors are Ali Behrouz, who is affiliated with both Google and Cornell, Farnoosh Hashemi of Cornell, Vahab Mirrokni, and Adel Javanmard, who is at Google and the University of Southern California.
Both obvious fixes have a flaw. Retraining the model’s weights on new material tends to cause catastrophic forgetting, where picking up a new skill erodes an old one. Leaving the material in the prompt works only until the session closes, and then the model has nothing to show for it.
The sleep design splits the model’s life in two. While awake, it behaves as normal and holds new information in short-term memory. When it sleeps, it stops taking input and spends its compute on storing that memory properly. Google says the model unlocks memory blocks reserved in advance inside a Mixture-of-Experts layout (an architecture where only some specialist sub-networks run at a time) and updates its weights using practice examples it writes for itself. The underlying architecture and training rules stay as they were.
Storing the memory takes two steps. New, small expert blocks absorb what the short-term layers were holding, using a teacher-and-student training step, so older parameters are left alone. Then the temporary short-term parameters are wiped, which frees room for the next waking session.
The second half of sleep is called dreaming. The model generates its own synthetic training samples, and the router that picks which experts fire is told to choose unusual ones at random, which pushes the model to mix knowledge from unrelated areas. A filter based on gradients keeps the samples most likely to make the model better, and the model is fine-tuned on those.
The reported results span four areas. On translation into Manchu and Kalamang, two languages the model had not seen, Google says performance barely dropped across several stages of learning, while in-context learning, Cartridges, and supervised fine-tuning forgot badly. On the BABILong benchmark the method scaled almost perfectly to 10 million tokens, ahead of GPT-4 and Titans. On factual updating it scored 48.9% in the single-task setting and 46.2% when tested after continued pre-training, ahead of SEAL. In math it reached 79.2% on AIME 24, 69.0% on AIME 25, and 46.1% on HMMT 25.
Google’s post is the only source for these figures, and it does not name the size of the model tested, say how long a sleep cycle takes or what it costs in compute, nor does it address what happens if the model consolidates something wrong. A system that rewrites its own weights from its own invented examples needs an answer to that last one before anyone lets it near customer data.
The business stake is where knowledge lives. Teams now keep models current with retrieval systems and long prompts, which are crude but auditable line by line. A model that absorbs a company’s daily material into its weights would be cheaper to query and much harder to inspect.
If the 10 million token result holds up outside Google’s lab, the retrieval pipelines many teams maintain to keep models current start to look like a fallback rather than the default.
Reported by Google Research on its research blog; the post carries no publication date.
