Why Large Language Models Fail at Tabular Prediction

A new study by Garnelo and Czarnecki identifies why large language models consistently underperform on tabular data prediction tasks compared to classical machine learning methods. Through systematic experimentation, the researchers found that LLMs' accuracy dramatically decreases as data dimensionality increases—a pattern opposite to classical baselines—while other hypothesized failures like noisy data or tokenization issues were ruled out. The findings suggest LLMs' capability for tabular tasks fundamentally dissolves in higher dimensions in ways that differ from simple noise corruption, though the exact internal mechanism remains unclear.