Blackwell FP4 attention reaches 2.13x speedup with direct probability quantization
Researchers have cracked a fundamental bottleneck in low-precision attention on Blackwell GPUs. While FP4 tensor cores promise speed, softmax conversion and memory dependencies had negated gains until now. Direct-P bypasses traditional softmax by mapping scores straight to quantized probabilities, delivering 2.13x forward throughput over bfloat16 on GB200. The causal variant reconstructs probabilities from quantized keys and queries, using FP8 gradients to accelerate single-GPU 8B-parameter training by 1.14x. This matters because attention remains the computational bottleneck in LLM inference and training, and hardware-aware quantization that preserves end-to-end speed is rare. Distributed training variants maintain FP8 precision throughout, suggesting a path toward production-grade low-bit attention at scale.
Modelwire context
ExplainerThe key innovation isn't just FP4 speed, but that Direct-P sidesteps the softmax bottleneck entirely by mapping attention scores directly to quantized probabilities without the intermediate precision loss that has historically killed low-bit attention gains on real hardware.
This work sits in tension with the quantization damage research from early September, which found that precision loss spreads across half the network rather than concentrating in specific layers. Direct-P suggests a different path: instead of trying to preserve precision globally or layer-by-layer, accept quantization early and design the operation itself to tolerate it. The causal variant's use of FP8 gradients also echoes the broader pattern from recent papers like Para-Pipe and GLM 5.3 Flash, where selective precision (not uniform low-bit) paired with hardware-aware design delivers real throughput without sacrificing convergence.
If Hugging Face or vLLM integrate Direct-P into their inference kernels within the next two quarters and report sustained 1.8x+ speedups on real LLM serving workloads (not just synthetic attention benchmarks), the technique has crossed from research to production viability. If the gains evaporate below batch size 8 or on models under 13B parameters, the result is too narrow to reshape inference infrastructure.
Coverage we drew on
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsNVIDIA · Blackwell · GB200 · FlashAttention · FP4 · FP8
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. arXiv cs.LG originally reported this story as “Hardware-Aware FP4 FlashAttention-4”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.