Taxonomies boost LLM antisemitism detection recall at cost of precision

Illustration accompanying: You Frame It: How Conceptual Representations Shape LLM Detection and Reasoning about Antisemitism

Researchers tested how LLMs detect antisemitism when given different types of conceptual grounding at inference time, from definitions to taxonomies to examples. Fine-grained taxonomies boosted recall substantially but hurt precision, while larger knowledge bases provided no measurable gains. Post-Holocaust antisemitism remained the hardest case across four state-of-the-art models. The work reveals a core tension in prompt engineering: richer semantic framing doesn't always translate to better reasoning, and domain-specific phenomena may require fundamentally different detection strategies than generic classification tasks.

Modelwire context

Explainer

The study's core finding cuts against conventional prompt engineering wisdom: adding taxonomic detail improved recall but degraded precision, suggesting that more structured semantic information can actually push models toward false positives. This inversion matters because it implies antisemitism detection may require constraint rather than enrichment.

This connects directly to the emotion classification work from early July, which found that fine-grained taxonomies (13-class emotion tasks) exposed significant performance gaps across frontier models. Both papers reveal a shared pattern: when you ask LLMs to reason about culturally or semantically loaded categories with precision, the models hit a wall. The financial knowledge graph study from the same period reinforces this: even with grounded inputs, hallucination detection remained unreliable. The implication is that domain-specific phenomena like antisemitism or emotion may require fundamentally different architectures than generic classification, not just better prompting.

If the researchers test their taxonomy-based approach on a held-out dataset of contemporary antisemitic content (not in training), and precision remains degraded relative to simpler prompts, that confirms the finding isn't an artifact of prompt brittleness. If precision recovers on a different model family (e.g., open-source vs. closed), the issue is model-specific rather than architectural.

Coverage we drew on

Quantifying the Affective Gap: A Zero-Shot Evaluation of LLMs on Fine-Grained Emotion Taxonomies · arXiv cs.CL

This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.

MentionsLLMs

Read full story at arXiv cs.CL →(arxiv.org)

Modelwire Editorial

This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.

Our mission How we write

Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “You Frame It: How Conceptual Representations Shape LLM Detection and Reasoning about Antisemitism”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.