REVA: Reusable Evidence View Aggregation for Context-Efficient RAG Serving

2609.11209 ↗ · 2026-09-10 · cs.LG, cs.CL, cs.IR

Tuan Nguyen, Qiran Hu, Banruo Liu, Khoa D. Doan, Kok-Seng Wong, Fan Lai

Author's accepted manuscript. Accepted for publication in the 2026 IEEE International Conference on Data Mining (ICDM)

The finding

Headline claim

Mining the target LLM's own historical attention traces into reusable, document-keyed text views compresses RAG contexts while improving answer quality over existing compressors and cutting compression overhead 5.3 to 15.6 times with under 40 ms added latency.

The test

Four RAG benchmarks with modern LLMs (e.g., Llama-3.1-8B), comparing REVA against truncation and state-of-the-art compressors on generation quality, compression latency, and evidence-recurrence rates (85 to 92 percent of held-out queries retrieve at least one previously seen document).

Effect size & scope

1.0 to 5.8 quality points over existing compression advances, overhead reduced 5.3 to 15.6 times, under 40 ms added latency; gains are selective rather than universal, and depend on documents recurring across queries (score-coverage for held-out queries).

Limitations

Authors concede their geometric reuse bound concerns retrieval similarity, not attention similarity, and no evaluated query contributes to its own store, so benefit relies on sufficient historical coverage; cold-start documents fall back to truncation. Unstated: gains may not transfer to long-tail corpora with low document recurrence, to closed API generators where attention is unavailable, or across model upgrades that invalidate the score store.

Counterpoint

The headline comparison may be against compressors chosen to be overhead-heavy; truncation being competitive suggests benchmark quality deltas are small and possibly prompt- or dataset-sensitive; only four benchmarks, one 8B open model shown, and no evidence of behavior in a production multi-tenant corpus.

Economy implication

Little or nothing for aggregate AI economics yet; it is one systems result. If it holds in deployment, it slightly reinforces [3.1]-style pressure: serving-cost engineering (reuse of attention traces, cheaper RAG prefill) keeps the per-query cost of grounded LLM apps falling, favoring app-layer margins over token-seller volume.

Who this matters for

Technical Capability: Shows a new free-lunch direction for RAG serving: the generator's own attention traces are a reusable compression signal, and naive truncation is a baseline compressors often fail to beat.

Market & Valuation: Reinforces that inference serving cost per useful output is being engineered down fast, a headwind for API token revenue but a tailwind for RAG-heavy app margins.

Bears on (advisory · papers never write evidence)

[3.1] Inference cost per unit of capability is falling rapidly.

Abstract

Retrieval-augmented generation (RAG) improves knowledge-intensive large language model (LLM) applications by conditioning generation on retrieved documents, but longer contexts increase latency, key-value (KV) cache memory, and token cost. Post-retrieval compression can reduce this cost, yet existing compressors often operate independently for each query, rely on auxiliary models or rewriting, and introduce online overhead that can offset the benefit of shorter prompts. We revisit RAG compression from a data-mining perspective by aggregating historical query--document--model interactions into reusable evidence views. We first show that modern compressors have unstable gains over simple truncation and can add substantial inference-time latency. We then propose Reusable Evidence View Aggregation (REVA), a framework that mines the target generator's historical attention traces into a document-keyed, budget-agnostic score store. REVA maps token-level attention to readable word units, aggregates importance across repeated document accesses, and renders budget-specific plain-text views that preserve document order and the standard RAG interface. Across four representative benchmarks and modern LLMs, REVA improves generation quality by 1.0--5.8 points over existing advances, while reducing compression overhead by a factor of 5.3 to 15.6, adding less than 40 ms of latency.