Recreating Minecraft Is Not a Benchmark

The article argues that recreating Minecraft and other viral demo tasks have become poor benchmarks for evaluating AI model capabilities. These "demo-benchmarks" are visually impressive and easy for the public to understand, but they're finite problems that newer models quickly perfect, making them unreliable indicators of actual AI advancement.