Language Models Need Sleep

Researchers propose a "sleep-like" mechanism for transformer language models that periodically consolidates recent context into persistent fast weights before clearing the key-value cache, allowing the model to perform offline recurrent passes during sleep while maintaining fast inference speed. Testing on synthetic tasks and math reasoning problems shows that increasing sleep duration improves performance, particularly for tasks requiring deeper reasoning, addressing the attention mechanism's poor scaling with context length. This approach shifts computational burden from inference time to offline processing, enabling better long-horizon task performance without compromising latency during actual prediction.