Language models allocate representation space by token context, not category structure
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Researchers challenge the prevailing interpretation of neural collapse in language models, arguing that within-class variance reflects deliberate information allocation rather than incomplete convergence. Analysis across 14 models reveals a consistent pattern: token-level context dominates representational structure at 79-91%, while category-level information comprises only 4-12%, stable across a 100x parameter range. The work reframes next-token prediction as an imbalanced classification problem where weight decay naturally orders category norms by type frequency rather than occurrence mass. This finding reshapes how practitioners should interpret representation geometry and has implications for model scaling and regularization strategies.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The practical implication buried in the framing is that regularization choices, specifically weight decay, are silently shaping how models distribute representational capacity across token types, not just controlling overfitting. Practitioners tuning weight decay for generalization may be inadvertently restructuring what the model 'knows' about rare versus common tokens.
This connects directly to the VLM counting paper covered the same day ('The Count Is There, but Misaligned'), which found that models encode correct information but fail at readout. Both papers are converging on the same underlying question: when a model gets something wrong, is the problem a missing representation or a structural mismatch in how that representation is accessed? The neural collapse paper adds a scaling dimension to that question, showing the 79-91% context-dominance pattern holds across a 100x parameter range, which means the geometry isn't an artifact of small models that disappears at scale.
If subsequent work tests whether deliberately adjusting weight decay schedules to counteract frequency-driven norm ordering produces measurable changes on rare-token tasks, that would confirm the causal claim here rather than leaving it correlational.
This interpretation is generated from the summary above and available source metadata. Our methodology · Report an error
MentionsLanguage models
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Neural Collapse Is Forbidden: Information Floors in Language Models”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.