Research·arXiv cs.CL·13h ago

Quantized caches break RAG faithfulness despite maintaining accuracy

Quantizing key-value caches in retrieval-augmented generation systems cuts storage costs but introduces a hidden risk: responses can remain factually correct while losing grounding in source evidence. Researchers testing Qwen2.5-7B-Instruct found INT8 quantization preserves both accuracy and faithfulness, but INT4 compression causes over 90% of correct answers to become unsupported by their retrieved context. This gap between accuracy and faithfulness matters for production RAG systems where users need verifiable, evidence-backed outputs. The finding suggests that compression metrics alone are insufficient for evaluating retrieval systems, forcing practitioners to choose between efficiency and interpretability.

Modelwire context

Explainer

The paper's core finding isn't just that aggressive quantization degrades performance. It's that INT4 compression creates a specific pathology: the model can produce factually correct answers that are completely disconnected from retrieved evidence, making errors invisible to standard accuracy metrics.

This connects directly to the audit work on clinical AI scribes from earlier this month, which found that roughly one in three notes contained verified errors that clinician review failed to catch. Both papers expose the same underlying problem: systems can appear to work correctly on surface metrics while failing in ways that matter for high-stakes deployment. The quantization paper adds a new dimension: efficiency optimizations can introduce these failure modes silently. The chunking and query optimization work from the same period focused on improving what gets retrieved and how it's extracted, but neither addressed what happens when the retrieval system itself becomes unreliable through compression.

If Qwen2.5-7B-Instruct INT4 quantization shows the same 90% faithfulness collapse when tested on the full HotpotQA test set (not just a subset), and if other model families replicate this pattern, then INT4 becomes effectively unusable for any RAG deployment where evidence traceability is required. Watch whether production RAG vendors publish their own faithfulness audits in the next quarter; silence would suggest they haven't measured this yet.

Coverage we drew on

One note in three: a verified census of three deployed AI scribes, and the instrument that counted it · arXiv cs.CL

This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.

MentionsQwen2.5-7B-Instruct · RGB · HotpotQA

Read full story at arXiv cs.CL →(arxiv.org)

Modelwire Editorial

This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.

Our mission How we write

Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Faithfulness Is Not Free: Auditing Offline KV-Cache Quantization in Retrieval-Augmented Generation”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.