EvoHarmBench: Breaking Content Moderation with Iterative Human-Like Evasion

2608.27844 ↗ · 2026-08-28 · cs.CL

Ruijie Jian, Benlei Cui, Ting Ma, Haidong Ding, Kangwei Liu, Ziwen Xu, Longtao Huang, Hui Xue, Ziqiang Zhu, Junjie Li, Haiwen Hong

Accepted to the Findings of EMNLP 2026

The finding

Headline claim

Iterative, human-strategy-guided adversarial rewriting defeats leading LLM content moderators: after twelve rounds, average attack success rate under readability constraints reaches 80.3% across seven state-of-the-art LLM moderators.

The test

A dynamic benchmark built from 5,002 real-world evasion samples from Chinese platforms, organized into 229 semantic sub-clusters across five violation categories, used to iteratively optimize cluster-level rewriting strategies (rewriting, comparison, and reflection all run on DeepSeek-V3.2-Exp) against thirteen moderation backbones over 12 rounds, scoring ASR@Readable with human validation showing 83.8% precision.

Effect size & scope

SOTA LLM moderators average 80.3% ASR@Readable, ranging from 73.0% (Gemini 3.1 Pro) to 88.8% (GLM 5.1); billion-scale open models average 91.2%. Gambling/fraud (89.8% avg) and pornographic content (91.7% avg) are weakest categories; abuse is most defended (62.5% avg). Holds in Chinese-language platform contexts; authors note the challenge generalizes beyond Chinese but do not test it.

Limitations

Authors concede: benchmark instantiated only on Chinese platforms; ASR@Readable is validated on a small manually annotated subset (83.8% precision, with disagreements mostly underestimation); attack components use a single fixed backbone (though sensitivity checks are in appendices). Unstated: the attack pipeline uses a frontier-class LLM to defeat moderators, so it measures attack budget as much as defender weakness; defensive models were not fine-tuned against this specific iterative attack; category mix and cluster granularity drive results; no test of false-positive costs on benign content.

Counterpoint

This is a benchmark paper whose attack framework could be seen as simply showing that any LLM armed with feedback loops and 12 iterations can break keyword-obfuscated Chinese moderation, a near-tautology of adversarial dynamics. The Comparison Model and Reflection Model judgments are LLM-based, so ASR@Readable may partly reflect one LLM's judgments of another's outputs. Results may also reflect specific Chinese evasion norms (homophones, character splitting) that English-focused moderators were never tuned for.

Economy implication

Little direct economic evidence yet, but it complicates the business case for LLM-based safety moderation: if dynamic attackers defeat frontier moderators at 80%+ rates, platforms face ongoing arms-race costs in moderation spend, and trust-and-safety vendors selling 'LLM moderator' products face a measurable efficacy gap between static benchmark performance and deployed robustness.

Who this matters for

Technical Capability: Shows a concrete, measurable gap between static safety evaluation and dynamic adversarial robustness, indicating current LLMs do not robustly recover intent under human linguistic strategies.

Societal & Cultural: Platform moderation reliability under adaptive bad actors is directly challenged, with implications for how much automated screening can be trusted at scale.

Abstract

Existing evaluations of harmful content detection rely predominantly on static benchmarks, which struggle to reflect the interactive adversarial ecosystem of real-world content platforms where users continuously revise their expressions in response to moderation feedback. This mismatch creates a significant performance gap between offline benchmark scores and online deployment effectiveness. To the best of our knowledge, we present EvoHarmBench, the first dynamic adversarial evaluation framework for content moderation systems. The framework employs an iterative optimization loop that evolves evasion strategies at the semantic-cluster level, while simultaneously optimizing for evasion success and human readability. We systematically evaluate LLM-based defense models which are widely used in real world moderation systems. The evaluation covers 229 semantic sub-clusters across five violation categories, derived from 5,002 real-world adversarial samples collected from content platforms. Our experiments reveal substantial vulnerabilities even in leading commercial systems: after twelve optimization iterations, the attack success rate under readability constraints reaches 80.3% within SOTA LLM moderators. We will release the full benchmark data, evaluation framework, and code to encourage a shift from static benchmarking toward dynamic adversarial evaluation in content safety research.