VICBench dataset brings production-scale vulnerability data to security AI

The critical detail buried in scale: 100 verified CVE-linked commits is small by modern dataset standards, but the 252-line average patch length is what matters. Most prior vulnerability datasets use synthetic or heavily simplified code, making them poor predictors of whether a model will catch real exploits in production codebases.

This connects directly to the structural inequality analysis from earlier this month on underrepresented languages. Just as Bengali speakers face a 67:1 token deficit in training data, security-focused LLMs face a similar upstream problem: the datasets used to train them don't reflect the complexity and diversity of actual vulnerable code. VICBench attempts to fix the denominator (dataset quality) rather than just scaling the numerator. The hybrid human-expert and agentic annotation approach also echoes the cascaded NLP pipeline work from the same day, which showed how combining unsupervised and supervised methods surfaces signal that single-pass approaches miss.

If VICBench-trained models outperform prior benchmarks on real-world CVE discovery in the next 12 months (measured by false negative rate on unpatched production code), the dataset has genuine predictive value. If performance gains don't transfer outside the benchmark's three languages or fail on zero-day patterns, that signals the dataset remains too narrow despite its improvements.