Lost in Sampling: Assessing Lexical Reachability in LLMs via the Word Coverage Score (WCS)

The paper's sharpest contribution isn't the metric itself but the causal argument: Top-p and Top-k don't just filter noise, they systematically truncate probability mass from low-frequency but contextually valid tokens as a mathematical consequence of how thresholds interact with skewed distributions. That means the problem is reproducible and predictable, not random.

This connects directly to the cultural awareness paper covered the same day ('Probing Cultural Awareness in LLMs'), which found that models fail to produce culturally resonant phrasing even when they demonstrably know the relevant vocabulary. WCS offers a plausible mechanical explanation for part of that gap: the words may exist in the model's learned weights but get filtered before generation. More broadly, the inference-time framing echoes the RAG reading work ('Separating Semantic Competition from Context Length'), which similarly argued that a persistent failure mode traces to a tractable, post-training intervention rather than to fundamental model limitations.

The practical test is whether adjusting Min-p thresholds (the paper's preferred alternative) produces measurable WCS improvements on culturally specific or domain-narrow corpora without increasing factual error rates. If a follow-up benchmark shows that gain, the inference-time framing holds; if error rates climb proportionally, practitioners face a real diversity-accuracy tradeoff that the paper currently underweights.