Using LLM for confidence scoring is a bad idea

Large language models cannot reliably generate trustworthy confidence scores because they lack the underlying information and reasoning process that produces valid confidence. Unlike weather forecasters who base confidence on data and past patterns, LLMs generate numbers without access to the specific information needed to calculate proper confidence levels. Confidence scores can only be validated by checking them across many predictions—if a model claims 70% confidence, it should be correct approximately 70% of the time—but LLMs have no mechanism to ensure this calibration.