Superposition Is Not Necessary: A Mechanistic Interpretability Analysis of Transformer Representations for Time Series Forecasting
Mechanistic interpretability researchers have challenged a core assumption about transformer success in time series forecasting: that superposition and complex representational tricks are necessary. Using sparse autoencoders to decode PatchTST internals, the work reveals that shallow, narrow transformers match deeper variants on standard benchmarks, and that simple linear baselines like DLinear remain competitive not through architectural accident but through genuine representational efficiency. This finding reshapes how the field should think about model complexity tradeoffs and suggests the transformer's power in forecasting may stem from different mechanisms than those driving NLP dominance.
Modelwire context
ExplainerThe deeper provocation here is methodological: the paper borrows sparse autoencoders, a tool developed for decoding LLM internals, and applies them to time series transformers, suggesting interpretability techniques may transfer across domains more readily than assumed. That cross-domain application is the buried lede the summary underplays.
This sits in direct tension with the MIT study covered on May 3rd, which identified superposition as the mechanistic driver behind scaling laws in LLMs. That work argued superposition explains why adding parameters reliably helps. This paper argues the opposite holds in time series forecasting: superposition is absent, and shallow models match deep ones. Together, the two papers suggest superposition may be domain-specific rather than a universal property of transformer success, which is a meaningful constraint on how far the MIT scaling explanation generalizes. The local attention expressivity work from May 1st adds a third data point: transformer efficiency gains keep appearing in places where the standard complexity story predicts they shouldn't.
If follow-up work applying sparse autoencoders to other forecasting architectures (N-BEATS, Mamba-based models) finds similarly sparse, non-superposed representations, the case that time series is a structurally distinct regime from NLP becomes hard to dismiss. If those probes instead reveal superposition, the finding here may be PatchTST-specific.
Coverage we drew on
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsPatchTST · DLinear · sparse autoencoders · mechanistic interpretability
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.