PhantomBench: Benchmarking the Non-existential Threat of Language Models

The benchmark's design choice to ground fabricated entities in real domains is the detail worth dwelling on: models can't dismiss these prompts as obviously nonsensical, so they pattern-match to adjacent real knowledge and confabulate confidently rather than abstaining.

This is largely disconnected from recent activity in our archive, as we have no prior coverage to anchor it to. It belongs, however, to a growing body of work questioning whether RLHF-trained models have any reliable internal signal for the boundary between what they know and what they are constructing. That conversation has been running in parallel to capability scaling debates, and PhantomBench is notable for putting a concrete number (86% across 21 models) on a failure mode that critics have described mostly in qualitative terms. The scale of the test set, over 60,000 entities, makes it harder to dismiss as anecdotal.

Watch whether any of the 21 evaluated labs respond with targeted fine-tuning runs on abstention behavior and then resubmit scores to the benchmark within the next six months. Sustained high failure rates after that window would suggest the problem is architectural rather than a training data gap.