LLM security analyzer rediscovers 68% of real-world CVEs without frontier models
Source published ·Modelwire updated
Original coverage: arXiv cs.LG ↗·How Modelwire adds context
The development
Researchers have built HoF-Bench, a rigorous evaluation framework grounded in 95 real CVEs discovered by AISLE, an LLM-based security analyzer that has already identified over 280 vulnerabilities in production systems like OpenSSL and curl. The benchmark enforces strict validation: detectors must pinpoint the exact code path, root cause, and attack surface without access to CVE metadata or frontier models. A minimal LLM analyzer recovered 68 percent of vulnerabilities under this protocol, establishing a reproducible standard for measuring AI-driven security research and challenging claims that only large frontier models can perform meaningful vulnerability discovery.
Modelwire’s AI-generated summary of coverage from arXiv cs.LG.
Modelwire analysis
Skeptical readOur AI-generated reading of the wider context and the next developments to watch.
The paper establishes a reproducible evaluation protocol, but the actual finding is more modest than the framing suggests: a minimal LLM recovered 68 percent of vulnerabilities, not all of them. The implicit claim that frontier models are unnecessary for vulnerability discovery conflicts with the fact that AISLE (which discovered the 95 CVEs in the benchmark) was itself a scaled system.
This connects to the broader pattern from 'Lottery Tickets Are Not Deployment Tickets' (late July), which showed that lab-validated optimizations often fail in production. Here, the benchmark proves that smaller models can match frontier performance on a curated task, but doesn't address whether they generalize to undiscovered vulnerabilities or whether the 32 percent gap matters for real security workflows. The rigor is genuine, but the practical implication remains unclear.
If AISLE or a similar minimal-model system discovers novel CVEs in the next six months that pass the HoF-Bench validation protocol without human annotation, that confirms the benchmark's predictive value. If instead the 68 percent recovery rate holds only for the 95 known CVEs and new discoveries require larger models, the claim collapses.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.LG
Sparse models fail real-world deployment tests despite matched accuracy
A new study challenges the assumption that sparse models and lottery tickets can seamlessly replace dense incumbents in production. Rather than relying on accuracy metrics alone, researchers audited deployment-critical behaviors including calibration, out-of-distribution robustness, and class-level reliability across a broad panel of models. The finding matters because practitioners often treat model compression as a plug-and-play…
MentionsAISLE · HoF-Bench · OpenSSL · curl · GnuTLS
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.LG originally reported this story as “HoF-Bench: Rediscovering Real AI-Discovered CVEs Without Frontier Models”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.