Pingyu Wu, Weiming Zhang, Nenghai Yu
Reported safeguard results like refusal and attack-success rates do not by themselves bound how much harmful assistance a deployed LLM service still supplies; in the coded literature, one executed attack suffices to show residual harm remains, but only one coded claim rules out the worst case.
The authors build a formal framework (evaluation anchor, residual harmful-assistance functional) with proofs of sharp bounds, then apply a coding instrument to 198 distinct safeguard papers at two depths, recording which coordinates each paper supplies and what deployment conclusion follows.
Asymmetric evidence: of 152 wide-coded claims, 108 report an adverse value putting residual assistance above zero (lower bounds easy); of 24 depth-coded claims, only 5 supply evidence about what remains possible after the check succeeds, and only 1 claim rules out the worst case. Supporting citations include adaptive attacks exceeding 90% success against most of 12 defenses originally reporting near-zero, and 0 to 1 percent first-turn success rising to 5.4 to 14.0 percent after 15 adaptive rounds.
The framework holds anchor coordinates fixed, so real deployments with shifting threat models, outcomes, or utility targets fall outside any single bound. The authors present the coded counts as hypotheses about what evaluations should report, not as a meta-analysis of safeguard effectiveness. The coding depends on what papers chose to report, and the full text provided cuts off mid-framework, so coverage of the upper-bound machinery (three gates, composition) is only sketched.
A skeptic could note the exercise is largely definitional: the claim that a local benchmark score does not prove deployment safety follows almost from the definition of a bounded test set, so the framework may formalize a known point rather than produce new empirical findings. Coding quality and subjectivity in assigning coordinates across 198 heterogeneous papers is hard to audit, and the headline counts depend on the authors' own inclusion and stopping rules.
Little or nothing yet for AI economics in the next 12 to 24 months. If the framing is adopted, it could eventually raise the cost of safety assurance for deployed LLM products (more system-level evaluation, weaker marketing claims from benchmark scores), but this paper supplies no measurement of that cost or of adoption effects.
Technical Capability: Safety and red-team practitioners should care because it tells them exactly which additional measurement (post-check reachability, coverage, continuation) converts a benchmark win into a deployment claim, and when no reanalysis can substitute.
Regulatory & Legal: Regulators writing compliance or safety-case requirements for AI services get a checklist of what evidence a deployment-safety claim actually needs, beyond reported refusal or attack-success rates.