Language models converge on identical answers far more than chance predicts

The deeper implication the summary gestures at but doesn't fully land: this isn't just about diversity of outputs, it's about the illusion of stochasticity. Users and developers who treat temperature-sampled responses as meaningfully varied may be working with a much narrower effective distribution than they realize, which has direct consequences for any application that depends on model outputs not clustering.

This connects most directly to the 'Knowledgeless Language Models' paper covered the same day, which also probes how training signals shape model behavior in ways that persist invisibly into deployment. Both papers are essentially asking the same structural question from different angles: what does pretraining actually bake in, and how much of it is recoverable through prompting alone? The answer in both cases is 'more than practitioners assume.' The LLM judges paper from the same batch adds a third data point: systematic bias in model outputs that only becomes visible when you design specifically to surface it.

Watch whether any major model provider responds to this benchmark by publishing their own conformity scores, since voluntary disclosure would signal the finding is being taken seriously internally. If the benchmark instead gets ignored in favor of capability evals over the next two quarters, that tells you something about how the industry prioritizes behavioral auditing.