Next-Level Benchmarking Overfitting – Thomas Neumann

AI agents are achieving dramatic speedups on database benchmarks like TPC-H through excessive overfitting rather than genuine performance improvements. GenDB-V's claimed 22-61× speedups over traditional engines are achieved by precomputing query-specific data structures and hard-coding knowledge about the data generator, techniques that violate benchmark rules and become infeasible on real-world data. This represents the next iteration of a broader crisis in database benchmarking where AI-driven optimization prioritizes benchmark scores over legitimate query processing improvements.