Common Crawl's PDF truncation hides 63% of text from LLM trainers
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
A new analysis of Common Crawl's PDF corpus reveals severe measurement distortion in how training data is reported. The study finds that document-level statistics mask extreme token concentration: just 3% of PDFs contain half the text, while Common Crawl's truncation cap silently discards 63% of affected documents' content. Existing PDF extraction libraries recover only 1-11% of this lost material. This matters because LLM trainers rely on corpus statistics to estimate data quality and coverage, yet published metrics systematically misrepresent what models actually ingest. The findings expose a critical gap between advertised and actual dataset composition.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The paper's core contribution isn't just that PDFs are skewed (that's known) but that the field has been conflating two different units of measurement. Reporting '100M documents' tells you nothing about actual text volume when 97% of documents are near-empty, and truncation losses are invisible in document counts.
This connects directly to the evaluation methodology problem exposed in the clinical error detection paper from August 17th. Both stories reveal how standard metrics mask real-world performance gaps: there, F1 scores hid pairwise discrimination failure; here, document counts hide token concentration and silent data loss. The common thread is that the field reports what's easy to count rather than what matters for deployment. The clinical work showed models fail on the task structure that actually matters; this shows training data statistics misrepresent the substrate those models learn from.
If major LLM papers published after this date begin reporting token-level statistics alongside document counts for their PDF subsets, or if Common Crawl releases a corrected corpus accounting for truncation losses, that signals the field is updating its measurement practices. If papers continue reporting only document-level metrics through Q4 2026, the gap between advertised and actual data composition persists as a hidden variable in model comparisons.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
Clinical LLM benchmarks mask poor performance, study finds
Researchers have exposed a critical flaw in how the field evaluates LLMs for clinical error detection. Standard metrics like F1 score mask poor performance when tested on paired comparisons, the natural structure of clinical benchmarks where each erroneous note has a correct counterpart. Across 15 diverse models and 4 multilingual datasets, 13 fell below random…
MentionsCommon Crawl · CC-MAIN-2021-31-PDF-UNTRUNCATED · TeX
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Counting Documents Is Not Counting Text: Unit Bias in Web-PDF Corpus Statistics”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.