Faithfulness Is Not Free: Auditing Offline KV-Cache Quantization in Retrieval-Augmented Generation

2608.30996 ↗ · 2026-08-31 · cs.CL

Atta Ul Asad, Ahsan Bilal, Muhammad Ali, Muhammad Haseeb, Dean F. Hougen

Grounding Language Models: Learning Faithfully and Efficiently @ EMNLP 2026

The finding

Headline claim

Quantizing offline KV-caches to INT4 in RAG silently degrades faithfulness (grounding in retrieved evidence) even on answers whose accuracy is preserved, while INT8 is near-lossless.

The test

Controlled audit on Qwen2.5-7B-Instruct across RGB (300 examples) and HotpotQA distractor split (300 examples), at K in {1,3,5} retrieved chunks, 3,600 generations per dataset; cache precision (BF16 vs INT8 vs INT4) is the only variable, faithfulness measured by HHEM-2.1, DeBERTa-v3 NLI entailment, and a Claude Haiku 4.5 LLM judge (92% agreement on 50 hand-checked labels).

Effect size & scope

On the accuracy-preserved subset, LLM-judge faithfulness flips were 231 worsened vs 24 improved on RGB (p<10^-38) and 173 vs 31 on HotpotQA (p<10^-30), i.e., over 90% negative; INT4 accuracy flips 133 vs 10 (RGB) and 117 vs 24 (HotpotQA). INT4 also yields 6% (RGB) and 3% (HotpotQA) degenerate outputs at K=5 and cuts refusal rate from 0.513 to 0.317 at K=1. Degradation grows with distractor fraction (beta=0.22) and retrieval depth (directional only, p=0.12). Realized storage savings: 1.9x (INT8), 3.6x (INT4), below nominal 4x. Scope: one 7B model, two QA benchmarks, one retriever.

Limitations

Authors concede: single model family, two QA-style benchmarks, three retrieval depths, production retrievers and larger models untested; HotpotQA refusal-calibration confound forces reliance on the LLM judge at low K; H2 slope test underpowered (p=0.12). Unstated: LLM-judge reliability rests on only 50 hand-checked labels; faithfulness judges themselves may be sensitive to quantization-induced surface changes rather than genuine grounding loss; no cost/latency measurement of the audit itself; only accuracy-preserved subset analyzed for the headline claim, and the effect may shrink with stronger models.

Counterpoint

Small n (300 per benchmark) and a single small open-weight model; the headline effect depends heavily on an LLM judge whose prompt sensitivity is not stress-tested, with moderate agreement (r=0.44) between the two detector signals; the benchmarks are synthetic RAG setups rather than production retrieval; a vendor could reasonably choose INT8, shown near-lossless, and treat INT4 findings as an edge case; no demonstration that the faithfulness gap matters for downstream users, only for metrics.

Economy implication

Little or nothing yet for macro economics. The narrow implication is for RAG serving infrastructure: teams cutting inference cost via aggressive offline cache compression face a hidden quality risk that standard accuracy metrics miss, so a faithfulness audit step may need to be added before deployment, a modest engineering cost rather than an economics-moving result.

Who this matters for

Technical Capability: Engineers choosing KV-cache precision for offline RAG get a concrete finding: INT8 is safe, INT4 buys 3.6x storage at the cost of grounding, refusal calibration, and degenerate outputs that EM and F1 will not reveal.

Market & Valuation: Vendors selling compressed-RAG serving stacks may need to differentiate on faithfulness auditing, since accuracy-validated compression can ship a silently degraded product.

Abstract

Retrieval-augmented generation systems can precompute and store key-value caches of retrieved documents to avoid re-encoding context at every query. Quantizing these caches further reduces storage, but no prior work asks whether compression damages faithfulness, whether responses remain grounded in the retrieved evidence. Faithfulness and accuracy are not equivalent: a model can produce a correct answer that is no longer supported by the context it was given. We evaluate Qwen2.5-7B-Instruct under INT8 and INT4 quantization on RGB and HotpotQA, measuring both accuracy and faithfulness with a hallucination detector, NLI entailment, and an LLM judge. INT8 is near-lossless across both metrics. INT4 reduces accuracy and, more critically, even among answers that remain factually correct, over 90% of faithfulness changes are negative, i.e., accuracy metrics are blind to this regression. The harm grows under noisy retrieval and with more retrieved chunks. Faithfulness must be audited before compressed caches are deployed.