Taming Outlier Tokens in Diffusion Transformers

The key finding is not just that outliers exist, but that they stem from corrupted patch representations in the diffusion process itself rather than extreme activation values. This distinction matters because it suggests the problem is baked into how DiT processes noisy intermediate states, not a symptom of training instability that masking can fix.

This connects to the broader pattern of domain-specific failure modes we've seen in recent research. Like Anthropic's work on sycophancy showing that alignment can be inconsistent across contexts (spirituality vs. reasoning), this paper reveals that transformer robustness assumptions from vision tasks don't transfer cleanly to generative diffusion. The outlier problem in DiT also echoes the KV cache bottleneck in multimodal models from early May: both expose how architectural choices that work in one setting create unexpected constraints in another. Where vision transformers tolerate local attention windows, diffusion transformers appear to require different handling of token heterogeneity during the denoising loop.

If practitioners report that the proposed fixes (likely involving representation-level constraints rather than token masking) reduce inference latency on consumer GPUs without quality loss, that validates the diagnosis. If the same outlier patterns appear in other diffusion architectures (Flow Matching models, latent diffusion variants) within the next two quarters, it signals a systemic issue rather than a DiT-specific quirk.