Stanford’s Memorilla Packs Entire Document Collections Into 16 Tokens and Beats Full-Context Prompting, Pointing to a Cheaper Way to Give AI a Memory

Every company that has tried to build a chatbot on top of its own documents runs into the same wall: a language model only knows what fits in its prompt, and stuffing that prompt with more files makes every answer slower and more expensive. A research group at Stanford thinks it has a better answer. On November 14, the team open-sourced Memorilla, a small add-on module that reads an entire document collection and hands an unmodified, “frozen” large language model just 16 memory tokens, no matter how big the collection is, according to MarkTechPost.

The headline result is hard to ignore. On the FactKG fact-checking benchmark, a frozen Qwen3-8B model using Memorilla scored 84.33% accuracy, compared with 80.43% when it was given every document in its context window, while using 108 times fewer document tokens, the project’s GitHub repository states. The code, training recipes and model checkpoints are public under the permissive Apache 2.0 license.

What Memorilla actually does

Today, businesses usually choose between two ways of feeding a model private knowledge. Full-context prompting pastes everything in, which works until the history gets long and latency balloons. Retrieval-augmented generation (RAG) pastes in only the text chunks that look most similar to the question, which is cheaper but can miss clues that don’t match the query’s wording. Memorilla takes a third route: it learns, for each question, what information the whole store needs to contribute, and compresses that into a fixed set of vectors the model reads alongside the question.

Per MarkTechPost’s breakdown of the paper on arXiv, the pipeline has three steps:

• Encode once. A frozen Qwen3-Embedding-4B model embeds each document a single time, so adding new documents never forces old ones to be re-processed.
• Read per question. Sixteen learned “slots” attend to the question and the full document store through two cross-attention layers.
• Inject. Those slots are projected into the language model’s input space and placed in front of the question.

Only the memory module itself is trained, about 194 million parameters for the 8-billion-parameter decoder, and the full three-stage training recipe takes roughly 19 hours on four Nvidia H100 GPUs, the report says. The work comes from Jure Leskovec’s SNAP group at Stanford, with co-authors from the University of Toronto, the University of Pennsylvania and MIT.

The numbers, including where it falls short

All baselines in the paper ran on the same frozen Qwen3-8B, and the scores are self-reported by the research team, so they haven’t been independently replicated yet. With that caveat, the reported results include:

• An average score of 49.68 across eight benchmarks, ahead of xRAG (41.92), LLMLingua-2 (40.75) and the popular memory layer Mem0 (38.02), per MarkTechPost’s comparison table.
• 44.60% accuracy on PersonaMem-v2, a test of remembering details about a user across many conversations, versus 27.92% when the model was given the full context.
• Compression of up to 4,212 times on NarrativeQA, where 16 tokens stood in for 67,385 document tokens.
• A lift in an AI agent’s average success rate on TextWorld games from 18.6% to 24.2% when it used Memorilla to remember its own history.

It isn’t a clean sweep. Memorilla lost on three of the eight benchmarks. On PubMedQA it scored 59.00%, well behind LLMLingua-2’s 69.00%; those medical abstracts total only about 324 tokens, so there is little to compress. MemoRAG also edged it out on PersonalizationV4 and NarrativeQA, though it used far more document tokens to do so. MarkTechPost flags fine-grained factual recall as a weak spot.

Why speed is the real business story

The latency figures may matter more to operators than the accuracy scores. On a single H100, Memorilla answered in 0.74 seconds with a collection of 512 documents, roughly 128,000 tokens, while putting all those documents in the prompt slowed the model from 0.75 seconds to 19.9 seconds, about 27 times slower, according to the repository. With an optional precomputed cache, the team reports a question taking 8.1 milliseconds against 1 million documents and 33 milliseconds against 4 million on a single Nvidia B200, though that cache needs about 80 GiB of memory at the 4-million-document scale.

For any business paying per token to a model provider, or paying for GPU time to run its own models, the gap between sending tens of thousands of tokens and sending 16 is the gap between a pilot project and something that can scale to every customer conversation. Long-running customer-service histories, account notes, contracts and product manuals are exactly the kind of sprawling, growing collections that make full-context prompting painful and that simple similarity search can miss.

What to watch before betting on it

There are practical limits for anyone hoping to drop this into production next week. The quickstart needs a Linux machine with an 80 GB CUDA GPU, the first run downloads about 25 GB of models, and no quantized versions have been disclosed, MarkTechPost notes. Each checkpoint must also run with the exact decoder it was trained with, so teams standardized on a different model family would have to retrain the module. And because the results are the authors’ own, the next milestone is independent testing on real enterprise data rather than academic benchmarks.

Still, the approach points at a direction the industry has been circling: instead of buying ever-larger context windows, teach a small, cheap component to decide what the big model needs to know. If that idea holds up outside the lab, AI memory could become a fixed, predictable cost instead of one that grows with every new document and every new customer.

The bottom line

Memorilla is an early, open-source research release, not a finished product, but its core claim, better accuracy than full context on some tasks with a tiny, fixed token budget and a fraction of the latency, is the kind of efficiency gain that can change the economics of AI assistants. Teams running RAG pipelines or agent memory today have a new, freely licensed baseline to test against. More details are on the Memorilla project site.

Edited by Coron @QUE.COM
Website: https://QUE.COM Intelligence
Sponsored by: https://MAJ.COM AI Autonomous