Research·arXiv cs.CL·19h ago

Open-source LLMs generate less diverse outputs than training data shows

Researchers using information-theoretic methods have found that major open-source language models consistently generate outputs with lower entropy than their training data, suggesting they fail to capture the full range of plausible responses present during training. The study, which measures conditional entropy without requiring multiple reference outputs, reveals a systematic gap between model diversity and data diversity across OLMo, Pythia, and GPT-Neo. This finding has implications for practitioners relying on LLMs for tasks requiring genuine output variability, and raises questions about whether current training objectives adequately preserve the distributional properties of source corpora.

Modelwire context

Explainer

The study measures a specific gap (conditional entropy loss) without requiring multiple reference outputs, making the finding reproducible across models. But the core claim is narrower than it sounds: models compress diversity, not that they fail to learn from diverse data.

This connects directly to the September 2 finding on logical validity representations. That work showed models can encode sophisticated structure internally while failing to surface it behaviorally. This entropy study suggests a related but distinct problem: models may learn the training distribution but systematically collapse it during generation. The two findings together paint a picture of models that internalize training data richly but then constrain their outputs through decoding choices or learned preferences, raising a harder question about whether low entropy is a bug (poor training) or a feature (learned safety or preference).

If the researchers show that fine-tuning with entropy regularization or alternative decoding methods recovers training data diversity without degrading downstream task performance, that confirms the gap is in generation mechanics rather than learning. If entropy remains low even after such interventions, the problem lies upstream in the training objective itself.

Coverage we drew on

When Decodability Is Not Enough: Logical Validity Representations, Behavioral Dissociation, and Causal Tests in Language Models · arXiv cs.CL

This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.

MentionsOLMo · Pythia · GPT-Neo

Read full story at arXiv cs.CL →(arxiv.org)

Modelwire Editorial

This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.

Our mission How we write

Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Do Large Language Models Capture the Diversity in their Training Data?”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.