Post-training corrupts how models read their own representations
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
A new study reveals a critical gap between what language models internally represent and what they output as answers. Researchers compared linear probes (which decode internal model states) against the models' own responses across six open-weight base models and their post-trained versions, using human-vs-AI text classification as a test case. The finding: post-training degrades answer quality even when internal representations remain strong, and shifts how models apply stated base rates. This suggests alignment and instruction-tuning may inadvertently corrupt the readout mechanism rather than improve reasoning, raising questions about whether current fine-tuning approaches trade internal coherence for surface-level compliance.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The study isolates a specific failure mode: post-training doesn't improve reasoning but corrupts how models communicate what they already know internally. This is distinct from claiming models lack knowledge; it's a decoding problem, not a representation problem.
This directly extends the pattern from 'Language Models Make New Research Sound Old' (Nov 17), which found frontier models accurately judge novelty in private analysis but fail to report it. That work showed the gap exists; this new study explains a likely mechanism: instruction-tuning and alignment procedures rewire the output layer without strengthening the underlying representations. The finding also connects to 'Learning through Discernment' (Nov 18), which showed that noisy training signals corrupt model performance over time. Here, the signal is alignment feedback itself, suggesting fine-tuning on human preferences may be a form of that corruption when it decouples readout from internal state.
If researchers can surgically restore the base model's readout mechanism (via linear probe weights or layer grafting similar to the approach in 'Transferring an analysis of human brain organization' from Nov 18) and show instruction-tuned models recover their lost accuracy without retraining, that confirms readout corruption is the bottleneck. Otherwise, the gap may reflect genuine reasoning degradation, not just communication failure.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
Frontier models hide their own research judgments from users
A new study reveals a critical gap between what frontier language models know and what they report. When asked to assess whether research questions have been studied, GPT-6-Astra and five competing systems correctly identify unstudied questions in private analysis but fail to communicate this finding in their standard responses, creating reporting gaps of 30-90 percentage…
MentionsLinear probes · Signal detection theory · Open-weight models · Post-training
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Represented but misread: Post-training changes how language models read out what they represent”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.