Theory explains why neural networks beat kernel methods on compositional tasks
Researchers have quantified a long-standing gap between neural networks and their neural tangent kernel approximations on compositional tasks, establishing that NTK performance degrades exponentially relative to depth-aware architectures. The work introduces a formal dichotomy between Fourier complexity, which governs kernel methods, and architectural complexity, which governs learned representations in finite-width networks. This finding clarifies why practical deep learning outpaces kernel baselines on structured problems and provides theoretical grounding for architectural design choices in compositional domains.
Modelwire context
ExplainerThe paper doesn't just show NTK underperforms on compositional tasks; it formally proves the gap is exponential in depth and traces it to a fundamental mismatch between how kernels and finite-width networks represent functions. This explains the 'why' behind a phenomenon practitioners have observed but theory couldn't quantify.
This connects directly to the function-counting theory work from early July, which relaxed classical assumptions to explain why deep learning generalizes on structured data. Where that paper addressed data geometry, this one addresses architectural geometry: both are closing gaps between classical learning theory and what actually works in practice. Together they suggest the theoretical foundations for deep learning are finally catching up to empirical intuition, which matters for practitioners trying to justify architectural choices beyond 'it works.'
If researchers use this dichotomy framework to predict which architectural modifications (skip connections, normalization schemes, width scaling) will outperform kernels on new compositional benchmarks before those benchmarks are published, that confirms the theory has predictive power. If the exponential gap holds only for ReLU networks and vanishes for other activations, the result is activation-specific rather than fundamental.
Coverage we drew on
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsNeural Tangent Kernel · ReLU networks · Fourier complexity · architectural complexity
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. arXiv cs.LG originally reported this story as “A Function-Space Dichotomy for Compositional Learning: Exponential Sub-Optimality of the Neural Tangent Kernel”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.