Can LLMs Understand the Impact of Trauma? Costs and Benefits of LLMs Coding the Interviews of Firearm Violence Survivors

The study's most pointed contribution isn't a performance score but a structural question: inductive coding, which deliberately lets meaning emerge from the data rather than forcing it into preset categories, may be fundamentally at odds with how LLMs pattern-match against training distributions. That tension is largely absent from the summary.

This connects directly to the reliability problems surfaced in recent Modelwire coverage. The 'Diagnosing LLM Judge Reliability' piece from April 16 found that even when aggregate consistency looks high, a substantial share of individual documents contain logical inconsistencies in pairwise comparisons. That finding matters here because qualitative coding of trauma narratives is precisely the kind of per-instance task where aggregate accuracy masks the cases that fail. If one-third to two-thirds of documents show hidden inconsistencies in controlled evaluation settings, deploying those same models on interviews from vulnerable populations compounds the risk considerably. The 'Context Over Content' paper from the same period adds another layer: LLM judges prioritize contextual framing over actual content, which raises questions about whether these models are reading trauma narratives or reading signals around them.

Watch whether the researchers or an independent group replicate this evaluation on a larger, demographically varied survivor dataset. If error rates cluster around specific trauma disclosure patterns rather than distributing randomly, that would confirm the failure mode is systematic rather than incidental.