Pretraining Exposure Explains Popularity Judgments in Large Language Models

The critical contribution here is not just that popularity bias exists in LLMs, which has been assumed for years, but that OLMo's open training corpus finally makes it measurable directly rather than inferred. Prior work could only correlate model outputs with external popularity proxies; this study closes the loop by counting actual token exposure.

This connects most directly to the Q-DAPS work covered earlier today ('Question Difficulty Estimation for Large Language Models via Answer Plausibility Scoring'), which also grapples with separating surface-level signals from genuine model reasoning. Both papers are, at root, asking the same question: when a model behaves a certain way, is that behavior evidence of learned understanding or an artifact of training data composition? The ORCE paper on confidence calibration ('ORCE: Order-Aware Alignment of Verbalized Confidence') adds another angle, since miscalibrated confidence in popular entities could compound the bias this paper documents. Together, these three papers from the same day reflect a broader push toward diagnosing what LLM outputs actually represent before trusting them in production.

The real test is whether this methodology replicates on a closed-corpus model, even partially, using indirect frequency proxies. If researchers can approximate the same exposure-bias correlation without direct corpus access, the finding becomes actionable for practitioners who cannot audit their model's training data.