Attention Is Not What You Need

Researchers propose a Grassmann manifold-based architecture as an alternative to self-attention in sequence models, arguing that explicit attention, while expressive, is mathematically opaque. Their attention-free model achieves comparable performance to Transformers on language modeling and natural language inference tasks while offering a more interpretable, geometrically-grounded approach to neural reasoning. The method encodes token interactions as low-rank subspaces that propagate information through controlled deformations, providing linear scaling in sequence length for fixed rank.