Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results
Fragmented evaluation results across incompatible formats and frameworks have long hampered reproducibility and cross-model comparison in AI research. Every Eval Ever addresses this infrastructure gap by proposing a unified JSON schema and community repository to standardize how benchmark scores are recorded and shared. The initiative targets a real pain point for researchers and practitioners: divergent scoring methodologies produce incomparable results even for nominally identical tests, inflating costs and blocking systematic evaluation science. Standardization here could reshape how the field validates progress and compares systems, making evaluation data as portable and reusable as model weights.
Modelwire context
Analyst takeThe harder problem Every Eval Ever sidesteps is incentive alignment: labs that benefit from incomparable results have little reason to submit scores to a shared repository, and the proposal's success depends entirely on voluntary participation from the same organizations that currently control benchmark narratives.
This story sits directly upstream of several evaluation efforts covered this week. SIMMER (covered June 12) illustrates exactly the problem a unified schema would address: a benchmark measuring latent planning failures produces results that currently have no standard home or format for cross-study comparison. Similarly, LoSoNA's findings on social norm inference in group conversations would gain traction faster if scores could be stacked against other behavioral benchmarks without manual reconciliation. The pattern across recent coverage is that researchers keep building specialized benchmarks for underserved failure modes, each in its own format silo. Every Eval Ever is a bet that the field is finally ready to treat evaluation data as shared infrastructure rather than lab-specific output.
Watch whether any major lab (Anthropic, Google DeepMind, or a frontier open-weights project) formally commits to submitting results in the proposed schema within six months. Adoption by even one credible institutional contributor would signal the repository has escaped the fate of prior standardization attempts that stalled at the proposal stage.
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsEvery Eval Ever
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.