CHASM: Unveiling Covert Advertisements on Chinese Social Media

The harder problem CHASM targets isn't spam detection, it's posts that are structurally indistinguishable from organic content, requiring models to reason across image, caption, and platform-specific social cues simultaneously. The benchmark's focus on Rednote (Xiaohongshu) also means the evaluation demands cross-cultural fluency that most Western-trained multimodal models haven't been tested against.

CHASM belongs to a growing cluster of domain-specific benchmarks designed to expose gaps that general LLM evaluations miss. We covered a similar impulse in MADE (from mid-April), which targeted medical adverse event classification precisely because standard benchmarks failed to capture label ambiguity in high-stakes contexts. The pattern is consistent: researchers are building narrow, real-world-grounded datasets because broad capability benchmarks don't stress-test the failure modes that matter in deployment. CoopEval, also from mid-April, extended this logic to social behavior, finding that models optimized on standard evals still defect in game-theoretic settings. CHASM adds deception detection to that list of underexplored blind spots.

Watch whether any of the major multimodal model providers (Qwen, GPT-4o, Gemini) publish results against CHASM within the next two quarters. If scores cluster near random baseline on the most visually ambiguous ad categories, that confirms the benchmark is actually stress-testing something models can't currently handle rather than measuring a capability they already have.