Yongxi Zhou, Wenbo Ye, Yuanzhe Liu, Zihan Dong, Junwei Yao
8 pages, 1 table. Code, wrappers, and per-verdict labels: https://github.com/Yongxi-Zhou/safety-judge-robustness
Specific LLM safety judges, including the deployed Llama Guard 4, have judge-specific blind spots that fixed, content-invariant style wrappers exploit at rates far above the judges' own noise floors, even though most judges in the panel are robust to most wrappers.
600 JailbreakBench responses (300 harmful, 300 refusals, across 4 target models) x up to 7 wrapper forms x 8 judges, with paired McNemar tests, Holm correction, 5000-sample bootstrap CIs, a measured self-disagreement noise floor from three re-scorings, majority-of-three rescoring, and a 145-item two-annotator human validation; realized API cost under $5.
The token-refusal wrapper flips 19.9% of GPT-4o-mini's correct unsafe verdicts to safe (95% CI [15.0, 24.0], n=291, noise floor 0.5%; 18.2% under majority-of-three), but moves Claude only 0.4%; educational framing flips 12.3% of Llama Guard 4 harmful verdicts to safe (8.3% for token refusal); a pseudo-compliance prefix flips 100% of keyword-judge refusal verdicts to unsafe (n=287); gpt-oss-safeguard-20b is immune at <=1.2%, below its 1.5% noise floor; tone-only wrappers still flip up to 2.1% (3-4x noise floor on vulnerable judges); the effect lives in the judge, since a StrongREJECT-style grading prompt cuts the token-refusal attack roughly tenfold on the same model.
The authors concede effects are judge-specific, not universal; the headline 19.9% effect involves an assertion-adding wrapper whose 'style' framing is contestable; assertion-adding flips are 2-6x tone-only flips; wrapped inputs also destabilize the judge (self-disagreement rises from 0.5% to 10.0% on GPT-4o-mini), so some flips may be boundary noise rather than systematic preference; the four-model safety ranking is already unstable to sampling alone, so wrapper-induced leaderboard reversals cannot be confirmed; n=600 from one benchmark artifact set, two annotators for validation, ShieldGemma and WildGuard deferred to camera-ready, small overall scale (single dataset, binary verdicts only).
A skeptic would note the headline attack (refusal-then-comply) arguably changes discourse structure, so calling it content-invariant overstates the result; the leaderboard-stability finding is a pre-existing property of the artifact set, not caused by wrappers; the set of tested judges is small and API-served, results may not transfer to production prompt configurations or other guard models; the dramatic 100% flip is on a trivial keyword baseline, not a real judge; and the finding that most judges are robust could be the actual headline the title obscures.
Little direct economic significance in the next 12 to 24 months, except that reported AI safety-compliance numbers and jailbreak-attack benchmarks rest on judge infrastructure of widely varying reliability, which matters modestly for trust-and-safety tooling procurement and for the credibility of safety claims vendors make to regulators and enterprise buyers.
Technical Capability: Evaluation builders and guard-model deployers should validate their specific judge against content-invariant restyling, since immunity (gpt-oss-safeguard-20b) and severe vulnerability (Llama Guard 4, GPT-4o-mini prompts) coexist in the same class of tools.
Regulatory & Legal: Regulators treating jailbreak success rates or safety leaderboard rankings as evidence of model risk should know the underlying judge labels are demonstrably unreliable and ranking-unstable on this artifact set.