Beyond Retrieval Failure: Understanding Why LLMs Miss Evidence in Multi-Hop RAG

2610.21184 ↗ · 2026-10-15 · cs.CL

Jiaqi Wei, Wenxuan Huang, Chengdong Xu, Xiang Zhang, Wanli Ouyang

Accepted by DAI 2026 Oral

The finding

Headline claim

In multi-hop RAG, most apparent retrieval failures are actually search drift (the agent stops querying for evidence the fixed retriever could reach), and a training-free requirement register cuts missing-evidence failures by roughly a third, shifting the bottleneck to premature answering and abstention rather than retrieval.

The test

1,000-question samples from MuSiQue (primary), HotpotQA, and 2WikiMultiHopQA using the HippoRAG corpora; GPT-4o-mini as generator with all-MiniLM-L6-v2 and Qwen3-Embedding-0.6B retrievers; ExecutionRAG compared against direct inference, vanilla/CoT/IRCoT retrieval, and structured/agentic RAG baselines (HippoRAG 2, RAPTOR, LightRAG, etc.), with failure causes labeled per wrong answer against gold supporting passages.

Effect size & scope

On MuSiQue, wrong answers lacking a required passage fell from 382 to 234 (69% to 44% of wrong answers) with MiniLM and from 383 to 274 (66% to 48%) with Qwen3; missing-evidence failures rise from 62% at two hops to 81% at four hops. Critically, the authors concede the recovered evidence does not translate proportionally into total error reduction because premature resolutions and abstentions increase.

Limitations

The authors explicitly state the controlled reduction is evidence of a recoverable search-drift component, not a numerical lower bound on search-drift prevalence (baseline and treatment can fail on different questions); unreachable failures are not decidable in general; the register guarantees state preservation, not correct resolution judgments; and total accuracy gains are much smaller than the failure-category shift suggests. Unstated: single generator model (GPT-4o-mini), benchmark-only multi-hop QA, substring and LLM-judge metrics with known judging noise, and results shown here are only for MuSiQue.

Counterpoint

A skeptic would note the diagnostic result depends on an LLM self-labeling scheme and a judge model, that a scaffolded checklist beating a baseline agent is a well-known prompt-engineering effect rather than a discovery about retrieval, that only one benchmark's numbers are shown in the excerpt, and that gains in a niche failure metric may not survive stronger base agents that already track sub-questions.

Economy implication

Little or nothing yet for the economics of AI in the next 12 to 24 months. At most, it suggests that improving agent reliability on multi-step retrieval tasks is partly a cheap software-orchestration problem (no retraining), which if replicated could nudge the cost of building reliable RAG agents down without new models.

Who this matters for

Technical Capability: RAG/agent engineers get a concrete diagnostic: before blaming the retriever, instrument whether the search ever queried for the missing passage, and expect the next bottleneck to be evidence verification, not retrieval.

Bears on (advisory · papers never write evidence)

[1.3] Agentic reliability (multi-step completion without human correction) is improving materially, not just peak single-shot.

Abstract

Multi-hop retrieval-augmented generation (RAG) requires an agent to retrieve evidence for a sequence of dependent sub-questions. When a wrong answer lacks a required passage, the passage may be inaccessible to the retriever, or the agent may have failed to issue a query that would retrieve it. Standard retrieval evaluations conflate these two causes. We study how much apparent retrieval failure is instead caused by search drift: the search no longer tracks an unanswered sub-question. Holding the retriever, corpus, and generation model fixed, we introduce ExecutionRAG, a training-free requirement register that records open sub-questions and their supporting evidence. The register changes search guidance but not evidence access, so each gold passage it recovers certifies that the passage was reachable by the original retriever. On 1,000 MuSiQue questions, the number of wrong answers with unretrieved required evidence falls from 382 to 234 with all-MiniLM-L6-v2 and from 383 to 274 with Qwen3-Embedding-0.6B. These counts correspond to 69% to 44% and 66% to 48% of each system's wrong answers, respectively. Crucially, evidence recovery does not produce a proportional reduction in total error: premature resolutions and abstentions increase as missing-evidence failures decrease. The central finding is therefore a shift in the remaining bottleneck, from guiding search toward required evidence to verifying that the retrieved evidence is sufficient.