Aditi Patodiya
Submitted to IEEE Access
Prefix caching, when enabled by default in LLM serving engines, makes outputs depend on server cache history rather than the request alone, and coarser weight quantization sharply amplifies this divergence in multi-turn agentic workloads.
A paired experimental design holding model, seed, decoding parameters, request order, and batch size (one, serial) fixed, running an eighty-episode multi-turn tool-use workload (BFCL multi-turn base) twice per configuration with caching enabled and disabled, across two engines (llama.cpp, vLLM 0.11.0), four weight formats from 16-bit down to 3-bit k-quantization, on a single RTX 4090; plus isolation experiments varying execution order, one cache flag, and restored cache state, and a GSM8K single-turn bridge.
With caching disabled, repeated execution was bit-identical in every configuration, 0 of 800 episodes. With caching enabled, the agent's trajectory changed on 36.2 percent of episodes at 16-bit precision and 75.0 percent at four-bit. A single server-level prompt-cache flag moved run-to-run divergence by 37.5 percentage points; restoring cache state made each path reproduce on 40 of 40 items while still differing from each other on 14. Aggregate accuracy did not shift: individual items flipped correctness in both directions without directional bias. The moving-block bootstrap interval for the key configuration was 20.0 to 55.0 percent, wider than Wilson's 28.8 to 49.7, reflecting non-exchangeable episodes.
The authors concede episodes run in fixed order against one server and are not exchangeable, so intervals may be anticonservative; a driver version differed between two machine groups; the SGLang lane was dropped as budget-contingent before any collection; vLLM cache exposure was verified only from engine logs rather than per-request telemetry, and token comparison there was on decoded strings rather than token identifiers. Unstated: single independent researcher, single GPU and small model sizes (7B class) on consumer hardware, one primary workload of eighty episodes, and no evidence on whether divergence affects outcomes in commercial hosted deployments where kernels, batching, and cache configurations differ.
The effect is a known floating-point reduction-order artifact already documented in prior work, so the novelty is magnitude and quantization interaction, not discovery. The mechanisms shown are deliberately excluded from realistic serving: batch size one, greedy decoding, serial requests, single GPU, and open-source stacks are not how frontier APIs serve traffic, so external validity to production settings with continuous batching is untested. The benchmark outcome is unchanged in aggregate, a skeptic can argue this is a reproducibility curiosity rather than a reliability problem, and n of 80 episodes with dependence between them limits precision.
Little or nothing yet for the economics of AI directly. The practical signal is engineering-level: cache-induced divergence complicates benchmark replication and agent evaluation, and cheap savings from prefix caching (reported elsewhere at 41 to 80 percent cost reductions) come with a reproducibility cost that determinism modes could claw back only at a throughput price. Whether that trade-off moves enterprise costs or reliability claims in the next 12 to 24 months is not established here.
Technical Capability: Engineers and eval teams should care that turning prefix caching on, the default, makes identical requests history-dependent, so flaky agent evals may be a server artifact rather than model noise.
Market & Valuation: Anyone pricing caching-based inference savings against reliability SLAs should note the optimization is not behaviorally transparent, which could matter for vendors selling determinism or auditability.