|
Researchers introduce the Lattice of Latents, a new architectural enhancement for language models that maintains persistent horizontal computational states across tokens alongside the traditional vertical residual stream, enabling these latents to directly influence attention, computation, and decoding. On a 406M-parameter model trained on 20B tokens, the Lattice reduces validation perplexity from 15.41 to 14.40 compared to baseline Transformers, and achieves even greater improvements when combined with other enhancements, reaching 13.61 perplexity. The approach demonstrates that persistent per-layer state is an effective and complementary dimension for scaling language model computation, with consistent gains across multiple architectural variants including GDN, Mamba-2, and hybrid models.
Read Full Article →
← Back to latest