|
In 27,280 coding agent runs, 64% of inference spend went to runs that failed
Researchers conducted a large-scale benchmark testing 22 coding harnesses across 5 AI models on 62 tasks, finding that 64% of inference spending went to failed runs and that the best harness choice varies significantly by model—with cost per task rankings shifting by an average of 11 positions depending on which model was used. The study revealed four harnesses on the efficiency frontier (OpenCode v2, Pi, Codex CLI, and ZCode), and demonstrated that switching harnesses when switching models is crucial for optimizing both cost and performance.
Read Full Article →
← Back to latest