DiscoTrace: Representing and Comparing Answering Strategies of Humans and LLMs in Information-Seeking Question Answering

The more pointed finding isn't just that LLMs lack variety — it's that they systematically favor breadth, covering more ground rather than making the selective, rhetorically purposeful choices that characterize expert human answers. That's a structural bias, not a capability gap, and it has real consequences for how we interpret fluency as a proxy for quality.

This connects most directly to the reliability concerns surfaced in 'Diagnosing LLM Judge Reliability' from the same week, which found that LLM evaluators show logical inconsistencies in pairwise comparisons roughly a third to two-thirds of the time. DiscoTrace adds a complementary angle: if models also produce rhetorically homogeneous outputs, then using LLMs to evaluate LLM-generated answers compounds the problem — judges and generators may share the same structural blind spots. The IG-Search paper's focus on rewarding effective information retrieval is adjacent, but that work targets search query quality rather than how retrieved information gets organized into a response, so the overlap is limited.

Watch whether DiscoTrace's discourse act taxonomy gets adopted in any upcoming instruction-tuning or RLHF dataset construction pipelines. If a major fine-tuning effort cites it as a filtering criterion within the next six months, the framework has moved from diagnostic to prescriptive.