Finite differences outperform automatic differentiation in physics-informed neural networks
Researchers challenge the dominance of automatic differentiation in Physics-Informed Neural Networks by demonstrating that finite-difference methods can match or exceed AD performance while consuming less GPU memory and compute. The work exposes a critical correctness bug in standard PyTorch autograd when handling architectures with inter-sample dependencies like BatchNorm and self-attention, revealing that the conventional approach silently produces wrong gradients at PINN-relevant scales. This finding reshapes how practitioners should approach derivative computation in scientific machine learning, particularly for resource-constrained settings where the speed and memory gains of finite differences become material.
Modelwire context
ExplainerThe critical bug in PyTorch autograd (silent gradient corruption with BatchNorm and self-attention) is the story, not the finite-difference speedup. That bug has likely been silently producing incorrect physics solutions in production PINNs for years.
This connects to the interpretability and mechanistic correctness thread running through recent work. The August data attribution paper identified how fine-tuning can surgically corrupt model behavior in latent feature space; this PINN work surfaces a lower-level correctness failure in the differentiation layer itself. Both reveal that standard tools (fine-tuning pipelines, autograd) can silently fail in ways benchmarks don't catch. The difference: that work traced feature-level misalignment, while this exposes a gradient computation bug that affects any PINN relying on modern architectures.
If PyTorch ships a fix for the BatchNorm/self-attention autograd bug within the next two quarters, check whether the finite-difference speedup persists or shrinks. If AD remains slower even after the fix, that signals the memory overhead is structural, not bug-related, and finite differences become a permanent alternative for resource-constrained scientific ML.
Coverage we drew on
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsPhysics-Informed Neural Networks · PyTorch · automatic differentiation · finite-difference methods · BatchNorm · self-attention
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. arXiv cs.LG originally reported this story as “Derivative Computation in PINNs: Automatic Differentiation, Finite Differences and Beyond”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.