When Reasoning Hurts: Source-Aware Evaluation of Frontier LLMs for Clinical SOAP Note Generation

The finding isn't just that reasoning hurts here: it's that the degradation is benchmark-source-dependent, meaning the same model behaves differently across ACI-Bench, PriMock57, and OMI Health data. That source-awareness framing is the actual contribution, and it suggests the problem is about reasoning interacting poorly with specific documentation conventions rather than a blanket failure of chain-of-thought.

This connects directly to the faithfulness work covered the same day ('Investigating the Interplay between Contextual and Parametric Chain-of-Thought Faithfulness under Optimization'), which found that reasoning traces don't reliably reflect underlying model behavior. If reasoning chains are already weakly faithful to model internals, adding them to a structured output task like SOAP notes may introduce noise without adding genuine inferential value. The medical domain angle also links to the sparse autoencoder steering paper from the same period, which showed that post-hoc feature suppression outperformed naive inference in radiology report generation, reinforcing a pattern: in clinical documentation, less unconstrained generation tends to produce more reliable outputs.

Watch whether OMI Health or a comparable clinical NLP vendor publishes production metrics comparing reasoning-on versus reasoning-off configurations on live encounter data within the next two quarters. If the benchmark pattern holds in deployment, expect enterprise guidance from model providers to start recommending reasoning-disabled modes for structured clinical output tasks.