DASC: Decay-Aware State Compression for Hybrid Linear-Attention Serving

2608.30386 ↗ · 2026-08-31 · cs.LG, cs.AI

Yanqi Yu, Pingwei Sun, Jianchao Tan, Tao Zhang, Yuchen Xie, Xunliang Cai, Yao Liu

The finding

Headline claim

For hybrid linear-attention models, keeping only the long-retention-horizon recurrent-state units in prefix caches compresses state checkpoints 2.63x with near-full cache quality, cutting mean TTFT by 42.6% and raising input throughput 68.4% under fixed memory budgets.

The test

Analysis of decay structure in Gated DeltaNet and Kimi Delta Attention states, plus serving experiments in SGLang on Qwen3-Next-80B and Kimi-Linear-48B: cache-intervention ablations on RULER NIAH-S1 (100 examples per depth), weight-derived horizon correlations on 32 ShareGPT prefixes (1,152 GDN and 640 KDA prefix-layer comparisons), RULER retrieval, five end-task benchmarks, LoCoMo, and matched-HBM serving latency/throughput measurements.

Effect size & scope

Conservative DASC configurations compress KDA recurrent state checkpoints by 2.63x while staying close to full-cache quality; under fixed state checkpoint memory, mean TTFT drops 42.6% and input throughput rises 68.4% on Kimi-Linear. Weight-derived static horizons correlate with observed empirical horizons at median Spearman 0.801 (GDN) and 0.856 (KDA). Gains are modest for GDN (more uniform decay) versus KDA; aggressive thresholds lose accuracy, partly recoverable via suffix-refresh replay at extra compute cost.

Limitations

Authors concede the single-key RULER intervention exposes headroom only on simple recall, and that multi-key, aggregation, and reasoning tasks show states are not freely compressible; DASC-WR remains approximate and never recovers history before the replay window; the static horizon metric is only moderately correlated with state and readout magnitude (0.40 to 0.63). The text was truncated mid-experiments, so full quality tables, baseline comparisons, and statistical reporting were not visible for verification.

Counterpoint

Only two open-weight model families were tested, both with the same hybrid delta-rule lineage, so generality beyond GDN/KDA is unproven; 'close to full-cache quality' is qualitative without the truncated end-task numbers in view; the epsilon=10^-3 and a=-0.3 metric constants are hand-tuned; the serving wins depend on a specific fixed-memory-budget protocol in SGLang that other engines may not replicate; no comparison to state quantization alone was visible in the provided text.

Economy implication

Little or nothing yet for the AI economy in 12 to 24 months as a direct matter; it is a serving-efficiency result, but if hybrid linear-attention open-weight models keep scaling, cheaper long-context prefix caching is one small input lowering inference cost per unit of capability, which matters mainly through the open-weight cost curve rather than as a standalone economic claim.

Who this matters for

Technical Capability: Serving-systems and inference-engine builders get a concrete, weight-derived technique to raise cache residency for hybrid-attention models, the architecture class now used by trillion-parameter open-weight models.

Market & Valuation: If inference costs for long-context workloads fall a few tens of percent on hybrid models, it marginally strengthens the case that open-weight serving competes on price with frontier API vendors.

Bears on (advisory · papers never write evidence)

[3.1] Inference cost per unit of capability is falling rapidly.

[5.2] Open-weight models reach "good enough" parity for most economic use cases.

Abstract

Hybrid linear-attention architectures have recently scaled to large open-weight models, offering quality competitive with full attention while substantially reducing key/value (KV) cache growth. However, their in-place recurrent-state updates complicate cache management: prefix reuse requires state checkpoints alongside full-attention KV, while storing state checkpoints in full increases memory pressure, leading to more evictions and repeated prefill. By analyzing the decay structure of Gated DeltaNet (GDN) and Kimi Delta Attention (KDA), we find that different heads and channels retain prefix information over markedly different timescales, which we term \emph{retention horizons}. This variation suggests substantial compression potential in persistent state checkpoints. Building on this observation, we introduce \emph{Decay-Aware State Compression} (DASC), which derives retention horizons from model weights, selects long-horizon state units, and packs them into a ragged state checkpoint layout. To integrate efficiently with tensor-parallel inference engines, DASC furtherly balances compressed state checkpoints across TP ranks. On reuse, DASC either zero-fills omitted units or refreshes them from a bounded suffix with additional compute cost. Across retrieval and end-to-end reasoning benchmarks on Kimi-Linear, conservative DASC configurations remain close to full caching while compressing KDA recurrent state checkpoints by $2.63\times$. Under fixed state checkpoint memory budgets, the resulting capacity gains reduce mean Time to First Token (TTFT) by 42.6\% and improve input throughput by 68.4\%. At larger compression ratio, suffix refresh recovers much of the accuracy lost to more aggressive omission, at the cost of additional replay computation. Qwen with GDN exhibits a similar quality--efficiency trend, showing that DASC extends from channel-wise KDA to head-wise GDN.