Le Chen, Zishen Wan, Baixi Sun, Xiaolong Ma, Chih-Hsuan Yang, Feng Yan, Sheng Di, Franck Cappello, Rajeev Thakur
Coding-agent working memory is semantically heterogeneous, so memory-management policies tuned on calibration tasks may show apparent gains that fail to transfer to held-out tasks, and nominal token budgets do not equalize delivered context or management cost.
Typed-object accounting of 55 archived SWE-bench-Lite coding trajectories (1,350 in-context objects across eight repositories), plus two case-study policy evaluations: a 15-task calibration and 8-task held-out sweep of an object-aware compression heuristic, and a six-arm retrieval comparison on 8 of 24 task blocks (48 paired cells), with a small serving replay on an NVIDIA GB10 running Qwen2.5-Coder-32B under a 32,768-token limit. Primary endpoint was a process metric: repeated tool invocations with identical name and arguments.
On calibration tasks the object-aware policy reduced repeated calls by 1.633 versus FIFO (95% CI [-2.667, -0.700], Holm p=0.0146), but on 8 held-out tasks the FIFO contrast shrank to -0.500 with only two nonzero pairs (p=0.50) and no held-out contrast survived correction. In the retrieval study no arm contrast was significant (e.g., GA vs FIFO -0.375, CI [-1.250, +0.375]). Object shares: tool outputs 55.5% of pooled content volume, artifacts 28.3% volume but 38.9% of retention-weighted cost; compressed/raw size ratio 0.150 for artifacts versus 0.673 for tool outputs.
The authors concede the primary endpoint is a process-regularity metric, not repair success; verdicts are not official SWE-bench scores; the served model revision was unpinned and temperature unrecorded; only 8 of 24 task blocks completed all six arms; the retrieval study reuses development tasks; calibration-to-held-out non-transfer is suggestive, not proven overfitting; the structural compression probe lacks task IDs; and dollar/resource estimates are unresolved rather than billed measurements. Unstated: single-run-per-arm design means all effects confound generation variance with policy effects, and the hand-specified policy invites researcher degrees of freedom.
The paper is largely a negative-result and methods critique with tiny n (8 held-out tasks, one run per arm), a noisy discrete endpoint, and self-conceded instrumentation flaws (the access clock collapsing LRU toward creation order), so the headline 'calibration gains do not transfer' rests on a handful of nonzero differences that could easily be noise; the evaluation framework may be more valuable than any empirical finding.
Little or nothing yet in direct economic terms. The indirect signal is that agent-memory engineering (context management, serving limits) is emerging as a real cost and reliability layer, which matters for whether agentic coding tools convert benchmark performance into dependable enterprise workflows; this paper documents the measurement gap rather than quantifying any ROI.
Technical Capability: Builders of coding agents get a concrete checklist: report stored state, delivered context, management work, and process outcome separately, because nominal token budgets and calibration-task wins mislead.
Market & Valuation: Investors pricing agentic-coding ROI should treat benchmark-level agent performance as an upper bound, since memory-management and delivered-context gaps show real deployments can diverge from nominal capability.