VikingRAG: Accurate and Token-efficient Retrieval-augmented Generation over Structured Documents

2609.11390 ↗ · 2026-09-10 · cs.IR, cs.AI, cs.CL, cs.DB, cs.LG

Peiyuan Gao, Gaoyuan Zhang, Haojie Qin, Yahui Sun, Qianyi Zhang, Yunhao Zhang, Zeyu Wang, Wei Lu

The finding

Headline claim

A directory-aware RAG system that keeps document structure outside the prompt, reuses past retrieval traces as experience edges, and escalates to agentic multi-round retrieval only when needed matches the accuracy of state-of-the-art structure-aware RAG while using a small fraction of the tokens.

The test

End-to-end RAG experiments on real structured-document datasets comparing VikingRAG, VikingRAG-E, and VikingRAG-E+ against MoDora, BookRAG, KohakuRAG, and DeepRead, measuring answer accuracy and token consumption, plus ablations on experience edges, adaptive escalation, hyperparameters, corpus size, and backbone LLMs.

Effect size & scope

Base VikingRAG consumes 11.6% to 51.9% of the tokens of state-of-the-art methods at matched accuracy; with trace reuse and adaptive escalation (VikingRAG-E+) token costs fall to 5.1% to 32.5% while accuracy remains competitive. The exact numbers are abstract-level ranges; per-dataset results are not in the provided text.

Limitations

The authors do not state explicit limitations in the provided text. Unstated ones: accuracy is only described as competitive or matched, not consistently better; experience edges depend on accumulated query history so cold-start corpora get less benefit; the evidence-sufficiency check that gates escalation could itself misjudge sufficiency; results cover real datasets but of unspecified scale and domain mix.

Counterpoint

Token-percentage savings depend entirely on which baselines and prompt designs were chosen; DeepRead's high token cost comes from a design choice (serialized directories in the prompt) that is easy to beat. The 5.1% floor looks aggressive and could reflect generous trace-reuse hit rates on repetitive query distributions. Accuracy claimed as 'competitive' rather than superior suggests the tradeoff is cost for possibly slightly worse answers, and the sufficiency check introduces a new error mode. Code is released, which helps, but the abstract gives no per-query variance or significance testing.

Economy implication

If it holds outside the benchmark setting, this points to RAG serving costs falling substantially without accuracy loss, since tokens are the dominant marginal cost of grounded LLM answers at scale. That said, this is one systems paper from one group, so the honest read is directional: agentic retrieval can be made far cheaper by caching and selective escalation, which over 12 to 24 months would compress the per-query cost of enterprise knowledge-base products. It is not yet evidence about aggregate AI cost curves.

Who this matters for

Technical Capability: Shows that agentic multi-round retrieval, usually dismissed as token-hungry, can be made cheap enough for production by materializing traces and gating escalation.

Market & Valuation: RAG is a core enterprise AI spend; a 5x to 20x token reduction on grounded queries directly changes the unit economics of knowledge-base products.

Labor & Knowledge Work: Cheaper reliable RAG over structured corporate documents makes document-heavy analyst and support workflows more automatable, though this paper itself measures no labor effects.

Abstract

State-of-the-art retrieval-augmented generation (RAG) methods exploit document structures to acquire sufficient evidence, but often incur substantial token costs. To reduce structural-context tokens without compromising high RAG accuracy, we present {\sf VikingRAG}, a directory-aware semantic data management system that tightly integrates semantic and structural access to support structural-context-efficient, evidence-gap-driven multi-round retrieval. To further reduce token overhead of multi-round interaction, we materialize agentic multi-round retrieval traces as experience edges, and reuse these edges for similar queries, avoiding repeated multi-round exploration. To additionally reduce token costs when agentic multi-round retrieval is unnecessary, we introduce an adaptive escalation strategy that answers from one-round experience-augmented retrieval when the evidence is sufficient, and invokes agentic multi-round retrieval only otherwise. Experiments on real datasets show that the base system {\sf VikingRAG} matches high accuracy of state-of-the-art methods while consuming only 11.6\%--51.9\% of their tokens. With retrieval-trace reuse and adaptive escalation, token costs drop to 5.1\%--32.5\% while maintaining competitive accuracy and practical document-storage performance, showing the utility of this work for emerging AI knowledge bases.