LLMs Are Complicated Now

Modern large language models have become significantly more complex than earlier versions like Llama, incorporating numerous attention variants, mixture-of-experts routing, and multimodal encoders that rival the architectural complexity of recommendation systems. This complexity arose from the tension between needing continuous capability improvements and maintaining inference efficiency, making it increasingly difficult to test new components without partial optimization. The solution requires designing systems for composability from the start—like PyTorch's FlexAttention—rather than attempting to hand-optimize or auto-generate solutions without a verified baseline.