| |
How good are frontier models at physics?
A comprehensive expert re-evaluation of frontier language models on physics benchmarks reveals that reported low scores significantly understate their actual capabilities, with many "failures" stemming from flawed reference solutions, grader errors, and poorly specified questions rather than model errors. After correction by domain experts, GPT-5.6-Sol's performance jumped dramatically—from 47.3% to 78.7% on HLE-Physics and from 61.0% to 87.2% on CMT-Benchmark—approaching saturation on well-posed problems. The findings highlight that current physics benchmarks are fundamentally broken and indicate the need for more rigorous, expert-validated evaluations to properly assess frontier models' scientific reasoning abilities.
Read Full Article →
← More Science news