Why are large language models so terrible at video games?

Large language models struggle to play video games because they lack the real-time decision-making capabilities and sensorimotor feedback loops required for gameplay, despite being able to code games themselves. The models are designed for text-based tasks and cannot effectively process dynamic visual information or make split-second decisions in interactive environments. According to Julian Togelius, director of NYU's Game Innovation Lab, this fundamental mismatch between how LLMs process information and what gameplay demands is the core reason for their poor performance.