Towards a Linguistic Evaluation of Narratives: A Quantitative Stylistic Framework

The 23-book test set is quite small, and the professional-vs-self-published distinction is a proxy for quality rather than a direct measure of it. Clustering those two groups successfully tells us the features capture something real, but it does not confirm the system can rank narratives within a tier or detect subtler editorial problems.

This paper sits directly inside a cluster of coverage questioning whether automated text evaluation can be trusted at all. The April 16 paper 'Diagnosing LLM Judge Reliability' found that even high aggregate consistency scores masked logical contradictions in one-third to two-thirds of individual document comparisons, and 'Context Over Content' showed LLM judges can be gamed by framing alone. The framework here sidesteps those problems by using deterministic linguistic features rather than LLM judges, which is a meaningful design choice worth noting. That said, it trades one problem for another: hand-crafted features may be stable but they can miss dimensions of quality that resist formalization, something the Poetry Camera review from The Verge gestured at when noting the gap between measurable output properties and felt aesthetic value.

The real test is whether this feature set generalizes beyond the professional-vs-self-published binary. If a follow-up study applies the same 33 features to rank narratives within a single publication tier and achieves agreement with human editorial judgment above chance, the framework has practical legs. If it does not, it is a classifier for production context, not quality.