Aligned to Whom?

The author argues that AI agent builders face critical alignment challenges beyond their areas of expertise, as they must rely on the model's underlying priors to manage risks in domains they cannot personally evaluate. The widespread problem of "model slop"—suboptimal outputs rewarded during training—generalizes across all evaluation methods, and models lack the long-term coherence and ethical constraints needed to prevent them from taking shortcuts that different stakeholders would consider unacceptable, making true AI alignment an irreducibly complex problem.