Blackwell FP4 attention reaches 2.13x speedup with direct probability quantization
Source published ·Modelwire updated
Original coverage: arXiv cs.LG ↗·How Modelwire adds context
The development
Researchers have cracked a fundamental bottleneck in low-precision attention on Blackwell GPUs. While FP4 tensor cores promise speed, softmax conversion and memory dependencies had negated gains until now. Direct-P bypasses traditional softmax by mapping scores straight to quantized probabilities, delivering 2.13x forward throughput over bfloat16 on GB200. The causal variant reconstructs probabilities from quantized keys and queries, using FP8 gradients to accelerate single-GPU 8B-parameter training by 1.14x. This matters because attention remains the computational bottleneck in LLM inference and training, and hardware-aware quantization that preserves end-to-end speed is rare. Distributed training variants maintain FP8 precision throughout, suggesting a path toward production-grade low-bit attention at scale.
Modelwire’s AI-generated summary of coverage from arXiv cs.LG.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The key innovation isn't just FP4 speed, but that Direct-P sidesteps the softmax bottleneck entirely by mapping attention scores directly to quantized probabilities without the intermediate precision loss that has historically killed low-bit attention gains on real hardware.
This work sits in tension with the quantization damage research from early September, which found that precision loss spreads across half the network rather than concentrating in specific layers. Direct-P suggests a different path: instead of trying to preserve precision globally or layer-by-layer, accept quantization early and design the operation itself to tolerate it. The causal variant's use of FP8 gradients also echoes the broader pattern from recent papers like Para-Pipe and GLM 5.3 Flash, where selective precision (not uniform low-bit) paired with hardware-aware design delivers real throughput without sacrificing convergence.
If Hugging Face or vLLM integrate Direct-P into their inference kernels within the next two quarters and report sustained 1.8x+ speedups on real LLM serving workloads (not just synthetic attention benchmarks), the technique has crossed from research to production viability. If the gains evaporate below batch size 8 or on models under 13B parameters, the result is too narrow to reshape inference infrastructure.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.LG
Quantization damage spreads evenly across LLM layers, not concentrated hotspots
Researchers mapped where quantization damage concentrates in compressed language models by systematically restoring precision layer-by-layer across nine models. The finding upends conventional wisdom: damage isn't localized to task circuits or computational bottlenecks, but distributed across roughly half the network. This challenges the assumption that precision budgets should target specific architectural regions, suggesting instead that gains…
MentionsNVIDIA · Blackwell · GB200 · FlashAttention · FP4 · FP8
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.LG originally reported this story as “Hardware-Aware FP4 FlashAttention-4”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.