LLMs used every day stay static after training: they know only what they saw up to a certain date and forget everything that happens within a session once it ends. Google Research tries to change this with an approach inspired by human sleep, which it calls the sleep framework.
The starting problem is well known. Updating model parameters with new data often triggers catastrophic forgetting, the loss of previously learned abilities. Relying only on the conversation context, instead, limits memory to the available text window, which empties when the session ends.
Wake and sleep
The proposal separates two phases. In the active phase, the model receives and processes inputs, storing temporary information in fast memory components. In the sleep phase, it pauses processing new inputs and redirects its computations toward memory consolidation and self-improvement. During sleep, the model unlocks pre-allocated Mixture-of-Experts memory blocks and updates its weights using practice examples it generates itself.
Consolidation and dreams
The first phase, consolidation, moves fragile memories into lower-frequency memory blocks. The model adds low-rank experts that absorb new knowledge without overwriting older parameters. The transfer uses distillation from a teacher model and a reward tied to semantic similarity. Once it is complete, the high-frequency temporary parameters are reset, freeing capacity for future sessions.
The second phase, dreaming, is a self-improvement loop. The Mixture-of-Experts router randomly activates non-standard experts, forcing the model to mix different domains of knowledge. The model generates synthetic training samples, filters them with gradient-based selection, and refines the result with reinforcement learning, checking whether the samples improve performance on downstream tasks.
Google argues the method has a theoretical basis. Consolidation is read as a low-rank compression problem: the model does not copy every detail, but keeps the directions that preserve both its own behavior and the meaning relevant to the task.
Image: Google Research — https://research.google/blog/the-sleep-framework-a-new-way-for-llms-to-retain-knowledge-in-continual-learning/
Why it matters
The idea fits a line of work Google Research has pursued for some time on continual learning, the ability of a model to acquire new skills over time without losing old ones. The same group presented the Nested Learning paradigm in November 2025, aimed at the same underlying problem.
In practical terms, the work aims at models that stay up to date without being retrained from scratch and that can retain information over long contexts. The preprint reports long-term memory up to 10 million tokens, a figure to be read as an experimental result rather than a feature already available in products.
Google describes the framework as an architecture whose code and model design remain unchanged: only the weights change. This detail matters because it makes the approach compatible with systems already in production. The work is still at the research stage, and no release date for commercial use has been given.
No comments yet. Be the first!