Anonymizing entities during pretraining reduces parametric recall in language models

The key inversion here is that this work treats parametric knowledge not as a resource to maximize but as a source of interference. Most pretraining research optimizes for more retained information; this work asks what happens when you systematically prevent that retention from forming in the first place, specifically by anonymizing named entities so the model never builds strong associations to retrieve later.

This connects directly to the evaluation problems surfaced in 'LLM Judges Can Be Too Generous When There Is No Reference Answer' from the same day. That paper showed judge models drift toward incorrect answers when they lack a ground-truth anchor, which is precisely the failure mode KLLM is designed to prevent at the generation level. Both papers are circling the same structural problem: models that rely on internalized priors rather than provided evidence produce outputs that are harder to verify and correct. The medical misconceptions paper ('Evaluating Large Language Models on Misconceptions in Multi-Turn Medical Conversations') adds another angle, showing that parametric confidence can actively resist correction across conversation turns.

The real test is whether KLLM-trained models maintain their context-grounding advantage on retrieval-augmented benchmarks like KILT or PopQA when the retrieved context is noisy or partially wrong. If accuracy holds under adversarial retrieval conditions, the tradeoff is genuinely useful for production RAG systems; if it collapses, the approach may just be shifting where errors originate.