HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data Mixtures
HERMES reframes data preparation for large language model training by decoupling the labeling system from the mixing strategy. Rather than forcing practitioners to choose a single semantic axis and granularity upfront, the method uses learned semantic transforms and residual vector quantization to create hierarchical, multi-resolution document codes. This enables dynamic adjustment of label granularity up to 130k cells without rebuilding annotations. The approach targets a persistent friction point in data-centric ML: practitioners currently rebuild entire label taxonomies when shifting between coarse and fine-grained mixing strategies. For teams optimizing training data composition, this substrate could reduce iteration cycles and unlock new mixing strategies previously constrained by static label architectures.
Modelwire context
ExplainerThe key insight is that HERMES solves a version mismatch problem: teams currently waste effort rebuilding label taxonomies whenever they want to shift between coarse and fine-grained data mixing. The paper doesn't just add granularity; it makes granularity a post-hoc dial rather than a pre-training commitment.
This connects directly to the broader data-centric ML efficiency push we covered in early July. The GRINCO paper on active learning showed how to reduce labeling overhead by treating redundant samples as equivalence classes rather than individual annotations. HERMES takes a complementary angle: instead of optimizing which samples to label, it optimizes the label structure itself to avoid re-annotation cycles. Both papers target the same bottleneck (annotation cost and iteration friction) from different directions. The multilingual corpus work (MultiSynt/MT) also shares the underlying problem: practitioners need flexible data infrastructure that doesn't force them to choose a single configuration upfront.
If teams using HERMES report that they iterate on mixing strategies 3+ times faster than baseline workflows without degrading model quality, the efficiency claim holds. If the 130k-cell ceiling gets hit in practice by large-scale training runs within the next 12 months, the method's scalability becomes the next constraint to solve.
Coverage we drew on
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsHERMES
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.