A Universal Context-Reuse Layer for Cross-Model KV Sharing

2608.30963 ↗ · 2026-08-31 · cs.LG, cs.AI

Yi Li, Dongming Jiang, Yi Zhao, Bingzhe Li

The finding

Headline claim

KV cache states from one LLM can be translated into a consumable form for a different model, so the target model can skip most of its own prefill on shared context while retaining acceptable accuracy or perplexity.

The test

Learned transport modules map source KV caches into target KV space for three source-target pairs: Qwen2.5-7B to Qwen2.5-1.5B (within family), Qwen2.5-1.5B to Gemma-2-2B (cross-family, similar scale), and Llama3.1-70B to Qwen2.5-7B (cross-family with scale gap). Evaluation uses LongBench2 accuracy, decoding perplexity, prefill cost, and measured latency, compared against native self-inference baselines for both source and target.

Effect size & scope

Within family, 7B-to-1.5B handoff raises LongBench2 accuracy from 27.59% to 34.48% over the native 1.5B baseline, still well below the 45.69% native 7B score. Cross-family at similar scale, 1.5B-to-Gemma-2 reduces target-side prefill cost by up to 67.05% at 4K context with perplexity close to native. The most heterogeneous case, Llama3.1-70B to Qwen2.5-7B, cuts measured latency from 899ms to 138ms but drops accuracy from 45.7% to 44.0%. Quality loss grows with source-target heterogeneity, and results cover only three handpicked pairs and a handful of benchmarks, so generalization is untested.

Limitations

The paper concedes the translated state is not the native KV cache and quality/efficiency tradeoffs depend on the pair; the evaluation is small in scope (three pairs, one long-context benchmark highlighted, single-context-length points for the latency claim). Unstated gaps include no end-to-end agentic or multi-model workflow test despite the motivation, no accounting of translation module training cost per pair, no memory footprint of storing/translating caches versus recomputing, and the text cuts off mid-setup so details like handoff training data and statistical rigor are not visible.

Counterpoint

Three curated model pairs on a few benchmarks is a proof of concept, not a systems result; a 1.7 to 2.6 percentage-point accuracy loss on the cross-family case may matter on real tasks; the latency win conflates prefill savings with total cost if translation overhead, cache storage, and transport are fully priced in; and learned translators must be trained per pair, which could erase the savings for short contexts or rarely reused pairs.

Economy implication

Little or nothing yet for the next 12 to 24 months as a deployed economics claim. If it matures, the direction is real: in routing, cascading, and multi-agent stacks, reuse of translated context could shave inference cost per unit of output and weaken the moat created by provider-specific prompt caching, but this paper only provides initial evidence.

Who this matters for

Technical Capability: This reframes KV caches from model-local artifacts to transferable state, which matters for anyone building heterogeneous multi-model inference infrastructure.

Market & Valuation: If context computation becomes portable across providers, prompt-caching discounts and other provider-specific inference lock-ins lose some pricing power.

Bears on (advisory · papers never write evidence)

[3.1] Inference cost per unit of capability is falling rapidly.

Abstract

Modern large language model (LLM) serving systems increasingly operate over repeated or shared context, yet each model typically performs its own prefill computation even when another model has already processed the same input. Existing KV-cache reuse mechanisms substantially reduce redundant computation within a single model, but generally assume that the producer and consumer of a cache are identical. We study \emph{cross-model KV sharing}, which translates the KV state produced by a source model into a representation that can be consumed by a different target model, including models that differ in scale, architecture, attention configuration, tokenizer, and model family. We evaluate the approach in both within-family and cross-family settings. For Qwen2.5-7B $\rightarrow$ Qwen2.5-1.5B, translated KV states improve LongBench2 accuracy from 27.59\% to 34.48\%, a gain of 6.89 percentage points over the native 1.5B baseline, while reducing handoff cost relative to native target prefill. For the cross-family Qwen2.5-1.5B $\rightarrow$ Gemma-2-2B setting, KV handoff reduces target-side prefill cost by up to 67.05\% at 4K context length while maintaining decoding perplexity close to native-model baselines. In a more heterogeneous Llama3.1-70B $\rightarrow$ Qwen2.5-7B setting, cross-family handoff achieves 44.0\% accuracy compared with 45.7\% for native Qwen2.5-7B inference, while reducing measured latency from 899ms to 138ms. These results provide initial evidence that KV states can serve as transferable computational representations rather than strictly model-local caches, and motivate \emph{context mobility} as a systems abstraction for reducing redundant prefill across heterogeneous LLM and multi-agent inference workflows.