PluRule: A Benchmark for Moderating Pluralistic Communities on Social Media

The benchmark's most underreported design choice is that it treats moderation as rule retrieval and application rather than binary harm classification, which is how actual human moderators work. That framing shift is what makes GPT-5.2's near-baseline performance genuinely diagnostic rather than just another capability shortfall.

The finding connects directly to a pattern visible across recent coverage here. The 'Artificial Intolerance' paper from May 17 showed that frontier LLMs inherit and amplify contextual biases embedded in training data, whether clinical notes or, by extension, the dominant community norms that likely over-represent large subreddits in any pretraining corpus. PluRule's 9-language, 2,000-community scope makes the same point structurally: models trained on majority-context data cannot reliably apply minority-community rules, even when those rules are explicitly provided. The multilingual annotation brittleness flagged in the Mandarin narrative transcripts paper from the same date adds a second pressure point, since PluRule's non-English violations are almost certainly where performance degrades furthest.

Watch whether any platform deploying AI-assisted moderation, Reddit being the obvious candidate given its role in the dataset, publishes internal accuracy figures against PluRule's rule-identification framing within the next six months. If they don't engage with this benchmark specifically, that silence is its own signal about how seriously community-level norm variance is being treated operationally.