The KV Cache Is the New Memory Wall

2609.30854 ↗ · 2026-09-25 · cs.DC, cs.LG, cs.PF

Tejinder Singh

28 pages, 12 figures, 12 tables

The finding

Headline claim

Long-context LLM decode is bound by KV-cache memory traffic rather than compute, and KV-cache compression techniques only yield end-to-end speedup beyond a hardware-specific crossover context length, with quality losses that differ sharply across five compression domains.

The test

A Systematization of Knowledge paper: closed-form roofline derivations of decode arithmetic intensity instantiated for H100, B200, and MI300X, plus evaluation of one representative method per domain (quantization, eviction, paging, prefix sharing, tiering) on standardized 128k-context workloads under one protocol that keeps derived speedup bounds separate from as-reported quality deltas.

Effect size & scope

Decode at batch size 1 runs at roughly 1 FLOP/B against a H100 ridge of about 295 FLOP/B. Llama-3-70B stores 0.33 MB of KV per token (42 GB at 128k, 328 GB at 1M tokens), exceeding one 80 GB H100 at 244k context. Architectural choice shifts the KV constant up to 12x (DeepSeek-V2 MLA at 0.07 MB/token). Compression savings approach the roofline bound beyond the crossover; below it KV compression yields negligible end-to-end speedup.

Limitations

The authors concede the published evidence base is mutually inconsistent (different workloads, hardware, quality metrics) and that quality and kernel interactions do not factor cleanly into their otherwise orthogonal byte-accounting model. They note quantization degrades below 4 bits and eviction fails discontinuously on position-sensitive tasks. Only one representative method per domain is evaluated; multi-die topology modeling relies on assumed per-die bandwidth partitions.

Counterpoint

This is a single-author SoK from a vendor (Dell), with derived speedup bounds rather than uniformly measured end-to-end results for most claims; the quality deltas are largely as-reported from prior papers rather than re-measured, so the comparative matrix inherits the same contamination it criticizes. The roofline model is standard and the headline arithmetic is uncontroversial.

Economy implication

Little or nothing for markets or labor directly. The concrete implication is that inference cost per long-context token is bounded by memory bandwidth, so efficiency gains will come from systems engineering (compression, paging, tiering) rather than chips alone, modestly supporting the case that per-token inference costs keep falling without new hardware generations.

Who this matters for

Technical Capability: Gives serving engineers and inference researchers an analytical frame (crossover context lengths, roofline bounds) for choosing among compression techniques instead of trusting isolated paper speedups.

Market & Valuation: Anyone modeling inference cost curves for long-context and reasoning workloads gets a first-principles basis for why bandwidth, not FLOPs, sets the cost floor.

Bears on (advisory · papers never write evidence)

[3.1] Inference cost per unit of capability is falling rapidly.

Abstract

Autoregressive LLM inference at long context is bounded by memory bandwidth, not arithmetic throughput, and the binding resource shifts from model weights to the Key-Value (KV) cache as sequence length grows. For Llama-3-70B in BF16, the 140 GB weight footprint exceeds the 80 GB HBM of a single accelerator, and one 128k-token sequence adds 42 GB of KV cache. Techniques that compress, evict, page, share, or offload KV state have proliferated, but reported gains use inconsistent workloads, hardware, and quality metrics, preventing cross-paper comparison. This SoK paper unifies the field analytically, with a protocol that strictly separates derived and reported claims. We derive closed-form arithmetic intensity as a decaying function of context length, parameterized by hardware topology for NVIDIA H100, NVIDIA B200, and AMD MI300X, including per-die bandwidth partitioning and the crossover lengths where KV traffic overtakes weight traffic. We classify the literature into five domains, quantization, token eviction, KV paging, prefix caching, and heterogeneous tiering, evaluating one method per domain at 128k context under a single protocol. The central finding is a three-regime structure: below a hardware-specific crossover, weight traffic dominates and KV compression yields negligible speedup; beyond it, KV traffic dominates and each domain trades quality for bandwidth savings approaching the roofline bound. Paging and prefix sharing are lossless but address capacity, not bandwidth. Quantization and eviction cut bandwidth directly, with degradation that accelerates below 4-bit precision and turns discontinuous for eviction on position-sensitive tasks. Tiering converts the bandwidth wall into an interconnect problem bounded by PCIe or NVLink rather than HBM. We close with design rules for selecting a compression domain given hardware, context length, and quality budget.