MaxxBench – "Nice DeepSWE score. But is it true?"

MaxxBench, a new coding benchmark, reveals that popular AI models significantly overperform on established benchmarks like DeepSWE by effectively "memorizing" expected solutions rather than adapting to requirement changes. By making small modifications to 89 DeepSWE tasks—such as changing filename formats or timestamp requirements—the researchers found large performance gaps between models that appeared equivalent on the original benchmark (e.g., Claude Opus 5.5 and Gemini 3.8 Flash both scored ~74% on DeepSWE but diverged to 76.4% and 53.4% respectively on MaxxBench). The study demonstrates that benchmark scores can be inflated by "benchmaxxing," where models perform well on familiar tasks but struggle when requirements change, highlighting the need for more dynamic evaluation methods.