SALUS automates detection of errors in NL-to-SQL benchmarks
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Benchmark quality has become a silent crisis in data-centric AI research. SALUS tackles this by automating detection of annotation errors in NL-to-SQL datasets, which are critical evaluation grounds for semantic parsing and database query models. The system deploys multiple LLM agents to generate candidate SQL outputs, then uses weak supervision and a generative label model to identify mislabeled examples without human review. This addresses a structural problem: corrupted benchmarks systematically underrate correct model behavior and obscure true progress. For researchers and practitioners, this work signals that benchmark auditing itself is now a first-class research problem, not a one-time curation task.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
SALUS doesn't just flag bad labels; it uses disagreement between multiple LLM agents as a signal for mislabeling, then applies a generative label model to rank confidence without human review. The key insight is that systematic annotation errors corrupt not just individual examples but the entire ranking of model capabilities.
This work sits directly alongside Argo-Bench (released Oct 1) and AutoDataBench (Sept 30), which both identified that existing NL-to-SQL benchmarks contain flawed answer keys and conflate data quality with algorithmic progress. Where those papers built better benchmarks from scratch, SALUS audits existing ones. The three papers together signal a recognition that benchmark corruption is now the primary bottleneck in measuring real agent progress, not model architecture or training data volume. The Agent Error Dataset (Sept 30) similarly mines failure patterns systematically rather than discarding them, suggesting the field is moving toward treating data quality as an active research problem rather than a one-time curation task.
If SALUS identifies significant error rates (>5%) in widely-cited NL-to-SQL benchmarks like Spider or WikiSQL when applied retroactively, and if those corrections shift published model rankings by more than 2-3 percentage points, that confirms benchmark corruption is masking real progress. If the tool sees adoption in the next six months for auditing new benchmarks before publication, that signals the field has accepted benchmark auditing as a prerequisite step.
This interpretation is generated from the summary above and available source metadata. Our methodology · Report an error
MentionsSALUS · NL-to-SQL · LLM agents
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “SALUS: Automated Auditing of NL-to-SQL Benchmarks through Weak Supervision of Multi-Agent Output”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.