How accurately calibrated is Jev?

Dylan Black tests whether Jev, TypeSafe's new "System One" classifier model, produces accurately calibrated probability distributions over multiple-choice answers. Using physics questions with well-understood answers (like the Maxwell-Boltzmann distribution), he evaluates how well Jev's confidence scores match actual correctness rates, finding that the model is poorly calibrated.