Memorilla: Latent Semantic Memory for LLMs

Memorilla is a new memory system that enables frozen large language models to efficiently access information from large document collections by compressing query-relevant data into just 16 fixed "memory tokens," achieving superior performance on multiple benchmarks while using 108x fewer tokens than full-context approaches. The system encodes documents into reusable embeddings and learns to retrieve and place query-specific information in the LLM's input space, with demonstrated benefits for both knowledge-intensive tasks and continual learning in agentic systems.