Working Memory Constraints Scaffold Learning in Transformers under Data Scarcity

The interesting inversion here is that the researchers aren't trying to overcome a constraint — they're deliberately imposing one. The argument is that limited attention windows force the model to generalize rather than memorize, which matters most precisely when you don't have enough data to memorize your way to good performance anyway.

This connects to a cluster of attention-efficiency work appearing on the site this week. The Stream-CQSA paper from arXiv on April 22 approached fixed attention windows as a hardware necessity, a way to prevent out-of-memory failures on long contexts. This paper flips that framing: the window isn't a workaround, it's a feature. AdaSplash-2, also from mid-April, pursued input-dependent sparsity to stay flexible. The cognitive-constraint paper argues the opposite direction has merit, at least in low-data regimes. These aren't contradictory, but they reflect genuinely different assumptions about what the bottleneck actually is.

The real test is whether these grammatical accuracy gains on BLiMP hold when the training corpus scales past 1 billion words — if the benefit disappears at that threshold, the technique is useful only for genuinely small-data applications like low-resource languages, not as a general architectural principle.