Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs

2609.11744 ↗ · 2026-09-10 · cs.DC, cs.LG

Joseph Kanichai, Tiziano De Matteis, Animesh Trivedi

The finding

Headline claim

External KV caching for LLM serving only pays off above a hardware- and workload-dependent break-even prefix size, and a scheduler-aware design with preloading (py-kvcache) can outperform existing systems like LMCache when it does pay off, but offers no benefit on real traces where average prefixes fall below break-even.

The test

Benchmarks of LMCache v0.4.7, llm-d, native vLLM KV Offload, and py-kvcache on vLLM (v0.16-v0.22), using synthetic long-document workloads, LongBench and SCBench, and Bailian production trace replays, on GPU (including H100) with GPU, CPU, and NVMe SSD tiers, measuring TTFT and component times.

Effect size & scope

At 80k tokens, py-kvcache from disk is 2.0x faster than LMCache (preload contributes 1.34x); in tiered configuration 1.23x faster than LMCache and within ~4% of native vLLM KV Offload. Preload lowers TTFT 1.66x vs no-preload at 40k tokens. On Bailian traces on an H100, average requests fall below break-even and external caching gives no TTFT benefit over GPU-only prefix caching. A single model (Llama 3.2 3B, 112 KiB/token KV) and small hardware set dominate results.

Limitations

Authors concede external caching is a setup-specific admission decision, not universally beneficial; break-even points are hardware and workload dependent. Unstated: single small model, limited GPU configs, one production trace source, comparisons against specific versions of competitors, and the proposed system is the authors' own so baseline tuning fairness is uncertain.

Counterpoint

This is an engineering characterization paper, not a general result: the headline speedups come from the authors' own system against one primary competitor (LMCache at v0.4.7) and a fork of llm-d; results could shift with newer versions, larger models, different SSDs, or different block sizes, and the 'no benefit on H100 for real traces' finding arguably undercuts the paper's own motivation.

Economy implication

Little or nothing yet for the macro economics of AI: this is an inference-serving cost optimization. At most it suggests inference cost per unit of output can be trimmed at the systems layer, and that cheap NVMe tiers can substitute for expensive GPU memory only for long-prefix workloads, a narrow slice of serving economics.

Who this matters for

Technical Capability: Inference-systems engineers get a measured map of when KV offload to NVMe helps versus hurts TTFT, including break-even thresholds and the value of preloading off the critical path.

Market & Valuation: Cloud and inference providers pricing long-context serving should note that GPU memory, not storage bandwidth, is often the binding constraint for real workloads on fast GPUs.

Bears on (advisory · papers never write evidence)

[3.1] Inference cost per unit of capability is falling rapidly.

Abstract

Prefix caching can reduce the time to first token (TTFT) of long-context LLM requests by reusing previously computed key-value (KV) states, but for short prefixes or fast GPUs, recomputation can be faster than loading from an external cache. We characterize this tradeoff in vLLM across GPU, CPU, and NVMe tiers using synthetic workloads, long-context benchmarks, production traces, and find that cache performance depends on transfer granularity, intermediate memory use, and when transfers enter the request schedule, not only on device bandwidth. These findings motivate py-kvcache, a vLLM KV Offload connector with asynchronous direct I/O, bounded shared staging, and scheduler-aware preloading, which starts disk reads while requests are still waiting, overlapping with compute. At 80k tokens, py-kvcache loading from disk is 2.0x faster than LMCache, with preloading contributing 1.34x. With GPU, CPU, and disk caching enabled, it is 1.23x faster than LMCache and within approximately 4% of the native vLLM KV Offload implementation. LongBench and SCBench show that these benefits extend to irregular prefix chains and multi-turn workloads. Bailian trace replays improve TTFT on a weaker GPU, but on an H100 the average request falls below the break-even point and GPU memory alone retains enough prefixes. External KV caching should therefore be treated as a setup specific admission decision. The py-kvcacheimplementation is available at: https://github.com/atlarge-research/py-kvcache.