Research·arXiv cs.LG·17h ago

Greedy decoding produces different outputs across numerical precisions

A new study reveals that greedy decoding in language models produces different outputs depending on numerical precision, even on identical hardware. Testing six models across three benchmarks showed 49-100% of prompts diverge between BF16 and FP16 formats, with single token differences cascading into entirely different trajectories. The root cause traces to how accumulated rounding errors interact with the margin between top logits at the output layer. This finding challenges assumptions about LLM determinism and has implications for reproducibility, deployment consistency, and the reliability of inference optimization techniques that trade precision for speed.

Modelwire context

Explainer

The study isolates precision as a source of non-determinism that compounds through autoregressive generation, not just a rounding artifact. This means two identical deployments on identical hardware can produce divergent outputs purely from numerical representation choices, making inference optimization decisions (BF16 vs FP16) consequential for output consistency, not just speed or memory.

This connects directly to the quantization exposure bias work from earlier this week. That research showed how sub-3-bit models drift from training trajectories during deployment; this paper reveals a lower-level mechanism: even full-precision models diverge based on how accumulated errors interact with logit margins. Together, they suggest that the gap between training and inference behavior runs deeper than just model capacity loss. The precision divergence also echoes the context noise problem from the long-context study, where small perturbations (there, nearby text; here, rounding) cascade into entirely different model trajectories.

If the same six models tested here show consistent divergence patterns when deployed in production systems (same prompts, same hardware, measured over weeks), that confirms this is not a lab artifact. Watch whether inference optimization frameworks (vLLM, TensorRT) begin documenting precision-specific output fingerprints as a deployment requirement, similar to how hardware verification now requires behavioral assertions.

Coverage we drew on

Train Where the Quantized Model Goes: On-Policy Distillation for Low-Bit Reasoning · arXiv cs.LG

This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.

MentionsBF16 · FP16

Read full story at arXiv cs.LG →(arxiv.org)

Modelwire Editorial

This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.

Our mission How we write

Modelwire summarizes, we don’t republish. arXiv cs.LG originally reported this story as “Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.