Yujin Zhou, Mingxuan Zheng, Chuxue Cao, Huang Yidan, Jiale Chen, Yike Guo, Sirui Han
EMNLP 2026 Main
Legal agents frequently reach correct final answers through hallucinated legal content or faulty agentic processes, and this Right-Answer-Wrong-Reason effect plus framework- and task-specific failure profiles are invisible to outcome-level evaluation.
A 3,414-instance expert-curated benchmark of Chinese-law questions (17 categories, 6 task types, ~45% subjective / 55% objective) built from four public Chinese legal benchmarks plus recent National Judicial Examination items, filtered to hard cases, annotated with a dual-layer taxonomy of 27 hallucination subclasses, and evaluated with an LLM-as-a-judge over full agent trajectories across 18 proprietary and open-source agents (LRAS legal agents and ReAct, Plan-and-Execute, LawThinker frameworks with shared tool suites).
The provided text reports no numerical main results; the excerpt cuts off during experimental setup, so magnitudes of the RAWR rate and hallucination frequency/density across the 18 agents are not verifiable from the given material. Scope conditions: Chinese law only, hard-case filtered items (all six filter models failed each retained closed-ended question), so absolute hallucination rates likely overstate real-world rates.
The authors rely on LLM-as-a-judge (Claude-4.6-Sonnet) for per-subclass verdicts, mitigated only by rubric anchoring and robustness checks in an appendix; the benchmark is Chinese-law specific yet not framed as such in the abstract; hard-case filtering biases toward failure; human verification of auto-generated rubric checklists has unknown coverage; results, code availability at release, and inter-annotator agreement numbers are not shown in the provided text.
A skeptic would note the judge is itself an LLM whose hallucination detection may be unreliable, that filtering to items six models all fail manufactures a maximal-failure dataset whose absolute numbers cannot be read as deployment risk, that the headline RAWR effect may partly reflect judge rubric gaming, and that no outcome-level baseline numbers are visible to confirm the claimed invisibility of the effect.
Little or nothing yet for AI economics in the next 12 to 24 months: this is a diagnostic benchmark, not an ROI or adoption study, though it warns that in a high-stakes vertical, correct-answer accuracy alone may materially overstate agent reliability, which could slow professional-services deployment if it generalizes beyond Chinese law.
Technical Capability: Shows that outcome-level benchmark scores miss trajectory-level failure modes in tool-using agents, meaning 'correct answer' metrics systematically understate hallucination risk in agentic systems.
Regulatory & Legal: For anyone drafting rules on AI in legal services, it argues that accuracy-only compliance testing is insufficient because procedurally unsound but correct answers pass.
Labor & Knowledge Work: Weakly relevant: if legal agents fail in legally subtle ways invisible to accuracy metrics, human lawyer review remains necessary, capping near-term automation of legal work.