Graph-augmented LLMs fail to use provided evidence, new diagnostic reveals
Researchers expose a critical gap in graph-augmented language models: simply embedding graph data into prompts does not guarantee the decoder can actually use it. Using HopQA, a diagnostic benchmark that measures shortest-path reasoning between nodes, the work demonstrates that existing systems fail on this bounded, unambiguous task despite having access to relevant graph evidence. The intervention framework separates three conditions to isolate where breakdowns occur: readable graphs, shuffled graphs, and no-graph baselines. This finding challenges assumptions underlying current retrieval-augmented and knowledge-graph-enhanced LLM architectures, suggesting that architectural or training changes may be necessary for models to reliably leverage structured external knowledge.62








