Heterogeneity in Formal Linguistic Competence of Language Models: Is Data the Real Bottleneck?

The more provocative implication buried in this result is diagnostic: if a tiny targeted data injection fixes most failures, then years of attributing grammatical weaknesses to transformer architecture may have been misattributing cause. The bottleneck was never the model's capacity to learn these structures, it was whether the training corpus contained enough examples to teach them.

This connects most directly to the pharmacoepidemiologic benchmarking paper from April 20, which raised similar questions about fine-tuning ROI when specialized models underperformed general-purpose ones. Both papers are probing the same underlying tension: how much of a model's apparent domain weakness is architectural versus a data curation problem. The 'From Fallback to Frontline' piece on LLM annotation also touched adjacent ground, noting that structural properties of training rather than domain knowledge often explain performance gaps. Taken together, these papers suggest a pattern worth tracking: targeted data interventions may be a more efficient lever than model scaling or architectural redesign for closing specific capability gaps.

The critical test is whether the same 1% synthetic injection approach holds on larger base models like GPT-2 Medium or beyond, since the gains could be specific to underfitting regimes that larger models don't occupy. If replication fails at scale, the finding is a small-model artifact rather than a general principle.