Instruction tuning amplifies confident hallucinations in language models
A new study identifies a counterintuitive failure mode in modern language models: instruction tuning and alignment procedures, designed to improve model behavior, paradoxically amplify confident hallucinations by 10 to 35 times compared to unaligned base models. Researchers across five model families found that post-training alignment drives models to express false information with high certainty on factual queries, particularly long-tail facts. This discovery undermines the assumption that uncertainty quantification alone can catch model errors and suggests alignment techniques may inadvertently calibrate models toward overconfidence rather than truthfulness. The finding has immediate implications for safety-critical deployments and raises questions about whether current alignment methods trade reliability for instruction-following.72
