Common Crawl's PDF truncation hides 63% of text from LLM trainers
A new analysis of Common Crawl's PDF corpus reveals severe measurement distortion in how training data is reported. The study finds that document-level statistics mask extreme token concentration: just 3% of PDFs contain half the text, while Common Crawl's truncation cap silently discards 63% of affected documents' content. Existing PDF extraction libraries recover only 1-11% of this lost material. This matters because LLM trainers rely on corpus statistics to estimate data quality and coverage, yet published metrics systematically misrepresent what models actually ingest. The findings expose a critical gap between advertised and actual dataset composition.
Modelwire context
ExplainerThe paper's core contribution isn't just that PDFs are skewed (that's known) but that the field has been conflating two different units of measurement. Reporting '100M documents' tells you nothing about actual text volume when 97% of documents are near-empty, and truncation losses are invisible in document counts.
This connects directly to the evaluation methodology problem exposed in the clinical error detection paper from August 17th. Both stories reveal how standard metrics mask real-world performance gaps: there, F1 scores hid pairwise discrimination failure; here, document counts hide token concentration and silent data loss. The common thread is that the field reports what's easy to count rather than what matters for deployment. The clinical work showed models fail on the task structure that actually matters; this shows training data statistics misrepresent the substrate those models learn from.
If major LLM papers published after this date begin reporting token-level statistics alongside document counts for their PDF subsets, or if Common Crawl releases a corrected corpus accounting for truncation losses, that signals the field is updating its measurement practices. If papers continue reporting only document-level metrics through Q4 2026, the gap between advertised and actual data composition persists as a hidden variable in model comparisons.
Coverage we drew on
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsCommon Crawl · CC-MAIN-2021-31-PDF-UNTRUNCATED · TeX
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Counting Documents Is Not Counting Text: Unit Bias in Web-PDF Corpus Statistics”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.