LLMs show consistent blind spots in evaluating creative work

The study isolates which dimensions of creativity assessment LLMs actually handle well (novelty detection) versus where they systematically fail (contextual judgment). This specificity matters because it suggests the problem isn't that LLMs can't evaluate creativity, but that they're blind to the factors humans use to separate interesting-but-useless ideas from genuinely impactful ones.

This connects directly to the broader pattern flagged in the arXiv cs.CL coverage from late July around LLM deployment opacity. Just as the pseudo-science study showed that model outputs depend heavily on configuration and vendor choice, this creativity research reveals that LLMs exhibit distinct evaluation biases depending on which model you deploy. Both papers point to the same underlying risk: organizations are outsourcing judgment calls to systems whose decision-making processes remain opaque and vendor-specific. The difference is scope. The pseudo-science work exposed how deployment choices shape epistemic validation in one domain; this study shows the problem generalizes to any subjective assessment task where context matters.

If the same six models show consistent divergence patterns when tested on a held-out creativity benchmark (e.g., a new dataset of real-world creative work with human consensus scores), that confirms the bias is structural rather than dataset-specific. If divergence shrinks after explicit fine-tuning on human judgments, that validates the paper's claim that calibration is possible but requires active intervention rather than occurring automatically.