Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?

The paper doesn't just propose using LLMs to judge poetry; it shows that having them adopt the author's interpretive stance produces more reliable assessments than generic evaluation prompts. This specificity matters because it suggests LLM evaluation quality depends less on model scale and more on prompt architecture that mirrors domain expertise.

This connects directly to the uncertainty-aware decision-making work from late June, which argued that LLM deployment should shift from 'better models' to 'better decision-making under ambiguity.' Poller operationalizes that insight for a concrete domain: poetry evaluation is inherently ambiguous, and the framework's success hinges on acknowledging that ambiguity by anchoring judgment to authorial intent rather than pretending objectivity exists. The same logic applies to peer review and tutoring tasks mentioned in that earlier paper. Where Poller differs is scope: it's domain-specific proof of concept rather than a general Bayesian framework, but both papers reject the assumption that confident outputs solve high-stakes subjective tasks.

If Poller's author-perspective framing outperforms human annotators on a held-out poetry corpus that wasn't used during framework design, the approach generalizes beyond literary tasks to other interpretive domains (legal brief evaluation, medical case review). If performance degrades when tested on poetry from cultures outside the training data's representation, that signals the method encodes evaluator bias rather than capturing domain structure, which would matter for any production deployment.