A frozen LLM, meaning a model that is not retrained, can use information outside its parameters: past conversations, long documents, the actions an agent has taken. The problem is cost. Many systems today turn that material into a large number of tokens the model must read on every request, with memory use and latency growing alongside the length of the history.
Researchers linked to Stanford, with affiliations also at Toronto, MIT and the Arc Institute, propose Memorilla, a latent semantic memory system. The principle is simple in form: each document is encoded once, much like a search index, and at query time the system retrieves the relevant information and translates it into a fixed set of 16 memory tokens placed in the frozen model's input space. The model always receives the same number of tokens, whatever the size of the archive.

Image: grahamslam — https://github.com/grahamslam/Memory-Color-Grid-MCG
The difference from the most common techniques lies in how much text stays in context. The paper reports a reduction in document-token usage of up to 4,212 times. In a test with 512 documents, the system runs 27 times faster. Against existing methods, Memorilla outperforms competing approaches on five of eight benchmarks, according to the authors.
The most important figure concerns the cost for the user. A 70,000-word legal document can occupy more than 100 gigabytes of internal representation in GPU memory, while the raw text weighs about 400 kilobytes. Shrinking that representation dramatically makes responses cheaper and faster, especially for those working with large document collections.

Image: TC407-api — https://github.com/TC407-api/titan-memory
The numbers remain those claimed by the authors in their work, published as an arXiv preprint and accompanied by a dedicated project page. No independent verification has been reported yet, and the comparisons concern benchmarks chosen by the research group. The work is nonetheless consistent with a direction already visible in other experiments on agent memory, such as those from Tencent and Google, where the goal is to keep token counts under control without losing useful information.

No comments yet. Be the first!