Counter-GEO-Bench: Evaluating Defenses Against Information-Distorting Generative Engine Optimization

Off-the-shelf safety guardrails (Granite Guardian, Llama Guard 3, NeMo Self-Check) fail to stop GEO-optimized misinformation from distorting generative-search answers, while a purpose-trained lightweight detector cuts attack success rate by roughly half with near-zero utility loss.

247 human-verified, quality-gated queries from GEO-Bench, each with paired information-preserving and information-distorting rewrites of one target document, run through a controlled hybrid-retrieval generative search harness with three open-weight victim LLMs (Gemma-4-31B, Qwen-3.5-35B-A3B, Llama-4-Scout-17B), an LLM judge (Claude Opus 4.6, human agreement kappa 0.74 to 0.87), and bootstrap confidence intervals.

Baseline guardrails reduce ASR by at most 3.2 percentage points (at most 5.7% relative), with Granite Guardian not statistically significant, and NeMo Self-Check anticorrelated (blocks clean, passes attacks). The authors' C-GEO Guard (0.18B DeBERTa-v3 classifier) reduces ASR by 47.6% relative (26.5 pp absolute) and 60.4% relative under GPT 5.5 rewrites, with near-zero FPR/accuracy loss. Results hold across three victim models but only within their own 247-query benchmark and controlled retrieval harness.

Authors concede small scale (247 instances, 3 victim LLMs, all open-weight mid-size), no adaptive attacker who knows C-GEO Guard exists, no open-set or unseen attack-class evaluation, and residual attacks remain. Unstated: training positives (293 docs) come from the same rewriting pipeline as the evaluation attacks, so the detector's gain may partly reflect template artifacts of the Sonnet 4.6 rewriter rather than general GEO-manipulation detection; retrieval harness (BM25+dense, 0.3/0.7) may not match production generative engines; LLM-judged ASR despite reported human agreement.

The flagship result (C-GEO Guard works) is train-and-test on the same generator and attack classes with shared templates and per-class centroids, so the 47.6% may be benchmark-internal overfitting; the guardrail-failure result is expected since safety taxonomies were never designed for fluent factual distortion; three victim models at 17B to 35B parameters say little about frontier victims that may resist misinformation via parametric knowledge; paired bootstrap on n=247 gives wide intervals; benchmark construction via one rewriter model creates a single style distribution attackers or defenders can overfit to.

Little or nothing yet for the AI economy in the next 12 to 24 months. The paper is a security-evaluation artifact. At most it flags a governance-relevant risk for generative-search products (retrieval-stage misinformation passing safety rails), which could matter for enterprise trust and platform liability if audits on production engines corroborate the ~half-of-queries contamination rates cited from prior work, but this paper itself measures no economic outcomes.

Technical Capability: Shows a concrete gap in the retrieval-to-synthesis stack: safety guardrails score zero against fluent informational misinformation, and the fix is a cheap 0.18M-scale classifier, so defense is tractable but not currently deployed.

Societal & Cultural: If generative search engines can be steered by ordinary-looking published pages, the information ecosystem inherits an SEO-style arms race where the output users see is distorted at near-benign cost to the attacker.

Regulatory & Legal: Guardrails that pass fluent misinformation suggest safety-taxonomy compliance does not equal misinformation robustness, relevant to any framework that treats guardrail deployment as a compliance marker.