Evaluating LLM-Driven Summarisation of Parliamentary Debates with Computational Argumentation

The core problem here isn't summarization quality in the general sense, it's that standard automated metrics (ROUGE, BERTScore, and similar) fail to detect when an LLM summary distorts the argumentative structure of a debate, even while scoring well on surface-level fidelity. Parliamentary text is adversarial by nature, meaning opposing positions must survive the summarization intact, which is a harder constraint than factual accuracy alone.

This paper sits squarely inside a growing cluster of work questioning whether automated evaluation of LLM outputs can be trusted at all. The story 'Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations' from mid-April found that even high-aggregate-consistency judges show logical inconsistencies in a third to two-thirds of individual documents. That finding applies directly here: if LLM judges can't reliably rank outputs, and standard metrics miss argumentative distortion, the evaluation layer for summarization is thinner than most practitioners assume. The computational argumentation approach in this paper is essentially proposing a domain-specific alternative to both.

The real test is whether this framework produces evaluations that correlate with human judgments on debates outside the training distribution, specifically non-English or non-Westminster parliamentary formats. If it does, it becomes a credible replacement for generic metrics in civic-tech applications; if not, it's a narrow instrument.