Attention to Mamba: A Recipe for Cross-Architecture Distillation

Researchers developed a two-stage distillation method to effectively transfer knowledge from Transformer models to Mamba (a State Space Model architecture) while maintaining performance. The approach uses principled initialization and linearized attention as an intermediate step, allowing the distilled Mamba model to retain nearly 99% of the original Transformer's performance on language modeling tasks, though with some gaps on tasks requiring in-context recall.