What Matters for Aggressive Decoding-Time KV Eviction? Temporal Aggregation and Ranking Preservation

2609.03515 ↗ · 2026-09-03 · cs.AI

Bo Zeng, Yu Zhao, Yefeng Liu, Zhihong Lu, Xuanfan Ni, Xintong Wang

Accepted to EMNLP 2026 Main Conference

The finding

Headline claim

At aggressive KV cache compression, the temporal aggregation rule (EMA smoothing) and preservation of score ranking, not the specific scoring function, largely determine eviction quality, and exploiting this ranking inertia allows refreshes to be reduced or eliminated with negligible quality loss.

The test

Decode-time KV eviction experiments on LongBench, LongBench-v2, and RULER across six open-weight backbones (Llama-3.1-8B, Qwen2.5-7B, Llama-3.3-70B, Qwen3-30B-A3B, plus Mistral-7B and Qwen2.5-14B stress tests) at 90% compression, comparing scorer variants (attention, value-norm, entropy, KeyDiff, learned scorer) under fixed EMA aggregation, plus throughput measurements on H100/H200 GPUs.

Effect size & scope

Order-preserving scorer variants produce near-identical retention sets (mean pairwise Jaccard 0.97) with average task-score deviation under 0.35 points, while a ranking-altering learned scorer diverges by 45.06 points. EMA beats single-step aggregation by +1.8 on LongBench and +21.6 on RULER. InertiaKV-Lazy yields 1.34-1.46x decode throughput over full-refresh InertiaKV; Score-Free decoding removes all post-first-step scoring with average quality change of +0.03. Scope: decode-time eviction only, aggressive (90%) compression, the evaluated sequential layer-temporal aggregation; not prefill compression or sparse attention.

Limitations

The authors concede the conclusion is scoped to the evaluated aggregation implementation, does not extend to scorers that substantially reorder the utility ranking, and cannot separate cross-step memory from the recurrence's implicit later-layer weighting. Adaptive alpha was a negative result. Score-Free effectiveness is an empirical observation about evaluated workloads, not a claim of universal first-step ranking optimality. Unstated: benchmarks are synthetic retrieval-heavy (RULER) or extractive (LongBench); open-weight models only; no proprietary model coverage; industrial affiliations (Ant, Alibaba) with no noted independent replication.

Counterpoint

Results at 90% compression may not transfer to the moderate-compression regime where production systems actually operate, and the scorer-invariance finding could partly reflect benchmark properties (synthetic needle-retrieval rewards stable early tokens) rather than a general aggregation principle. The throughput gain (1.34-1.46x) is relative to their own full-refresh method, not to the fastest baseline end-to-end, and decode-step speedups can vanish once attention compute dominates.

Economy implication

Little or nothing yet for the AI-economy debate directly; the paper's practical significance is that KV-cache scoring overhead is more removable than assumed, which modestly strengthens the ongoing inference-cost decline per token at long context, one input among many to serving margins over the next 12 to 24 months.

Who this matters for

Technical Capability: Systems researchers and inference engineers should care because it reframes KV eviction as an aggregation-and-refresh-frequency problem, meaning cheaper decode implementations may not require better scorers at all.

Market & Valuation: Cloud and inference providers competing on long-context serving costs should care because eliminating per-step scoring is a free-ish throughput lever that compounds with hardware-level price competition in inference.

Bears on (advisory · papers never write evidence)

[3.1] Inference cost per unit of capability is falling rapidly.

Abstract

Decoding-time KV cache compression research focuses heavily on designing better token scoring functions, while the temporal rule that aggregates scores across decode steps is often treated as an implementation detail. Under aggressive KV compression, we find that exponential-moving-average (EMA) aggregation makes approximately order-preserving scorer modifications largely indistinguishable at the eviction-set level. Value-norm and entropy variants remain highly correlated with attention and produce nearly unchanged retention sets, whereas KeyDiff, key norm, recency, and a learned scorer alter the ranking and degrade substantially. We associate this stability with the evaluated aggregation, which couples layer weighting and temporal retention. Building on this observation, we introduce InertiaKV, an EMA-based decoding-time eviction method, and InertiaKV-Lazy, its periodic-refresh variant, which yields 1.34-1.46x decode throughput relative to full refresh InertiaKV. We also study Score-Free decoding as a separate empirical operating point: it scores the full context once at the first decode step, freezes that ranking, and incurs an average quality change of +0.03 while removing all subsequent scoring. Across six open-weight backbones and the LongBench, LongBench-v2, and RULER benchmarks, the results identify temporal aggregation and ranking preservation as distinct, consequential design factors; they do not imply that scoring quality is irrelevant in general.