Can LLMs Beat Classical Hyperparameter Optimization Algorithms?

Classical hyperparameter optimization algorithms like CMA-ES and TPE consistently outperform pure LLM-based methods for tuning neural network hyperparameters under fixed compute budgets, primarily because avoiding memory failures is more critical than search diversity. The researchers introduced Centaur, a hybrid approach that combines CMA-ES's interpretable internal state with LLM guidance, which outperformed both classical and pure LLM methods and required only a 0.8B parameter model to be competitive. The findings suggest LLMs are most effective as complements to classical optimizers rather than replacements, as they lack the ability to track optimization state across trials while classical methods lack domain knowledge.