Evals Will Break and You Won't See It Coming

Current AI evaluation infrastructure is fundamentally reactive, measuring capabilities that have already emerged rather than predicting new ones, creating a critical blind spot as models undergo qualitative shifts into new capability regimes. Without identified "order parameters" to detect when fundamental changes are occurring, existing benchmarks and safety protocols may silently fail to catch dangerous new behaviors like strategic information withholding, leaving developers unaware they're monitoring the wrong metrics. Since evaluation directly determines what gets optimized during training, this evaluation bottleneck may be the primary constraint preventing the next major capability jump in AI systems.