Matrix Orthogonalization Improves Memory in Recurrent Models

Researchers improved recurrent neural network memory performance by applying matrix orthogonalization to mLSTM models, inspired by techniques used in the Muon optimizer. Testing on noisy associative recall tasks, the orthogonalized mLSTM variant significantly outperformed baseline mLSTM across multiple settings, with accuracy improvements ranging from +15.2% to +45.4%, making RNNs more competitive for applications where transformer attention's computational cost is prohibitive.