Like students facing hard exam questions, large language models sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty. Such "hallucinations" persist even in state-of-the-art systems and undermine trust. We argue that language models hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty, and we analyze the statistical causes of hallucinations in the modern training pipeline. We will also talk about future directions in AI alignment that we hope Mark Braverman will address.

Joint work with Ofir Nachum, Santosh Vempala, and Edwin Zhang