Dishu Yang, Qi Su, Hongbo Qin, Hansong Zhang
Agentic leaderboard contamination claims require channel-specific evidence, and under a fail-closed audit of nine HAL configurations plus a matched-control re-evaluation on SWE-bench Verified, susceptibility is broad (no channel closed, confirmed pipeline incidents in four configurations) but the behavioral file-localization gap is statistically inconclusive, so strong contamination or memorization claims are not warranted.
Stage I provenance audit across nine HAL configurations coded on three channels (training-time exposure, evaluation-time retrieval, pipeline/scaffold leakage) with revision-pinned, human-adjudicated evidence; Stage II re-evaluated two probes from Liang et al. on 100 matched same-repository SWE-bench Verified/control pairs (after symmetric leakage screening) using GPT-4.1 and DeepSeek-V4-Flash at temperature 0, with paired-bootstrap and repository-balanced uncertainty and a scorer validation gate against consensus human labels.
File-path Top-3 accuracy gaps were +0.070 for DeepSeek-V4-Flash (0.640 vs 0.570) and +0.100 for GPT-4.1 (0.710 vs 0.610), positive in all eight model-metric combinations, but the 95% intervals from both the prespecified paired bootstrap and the post-hoc repository-balanced analysis included zero, leaving the gap inconclusive; the reproduction scorer failed its validation gate (insufficient sensitivity for both models, both DeepSeek-V4-Flash firings on correct-gold comparisons were false positives), so reproduction contrasts remain exploratory. Scope: two models, 100 pairs, ten repositories, no retrieval tools, public controls not confirmed unexposed.
Authors concede no channel was coded closed under fail-closed rules (labels reflect susceptibility, not probability), temporal eligibility was unknown for every configuration, no configuration-scoped retrieval incident was established, the discoverability audit covers one service on one date and is not a prevalence estimate, controls may themselves be contaminated, DeepSeek-V4-Flash had no documented training cutoff, analyses are pointwise and unadjusted, and the results establish neither training-data membership nor benchmark-induced score inflation. Unstated: a single-issue collision and screening reduced 199 pairs to 100, and scorer sensitivity had no prespecified numeric threshold.
A fair skeptic could note the paper is largely a null result with heavy machinery: two models, one completion each, 100 surviving pairs, and a gap of +10 points on a 100-pair sample is consistent with a real but underpowered effect, so 'inconclusive' is not 'no contamination'; the framework's own coding had only 76.9% pre-adjudication author agreement; and the reproduction scorer failure may reflect the audit's strict scorer rather than the original probe's 5-gram approach, meaning the target claims of Liang et al. were not fully tested on their own terms.
Little or nothing yet economically: the paper is a measurement-methodology contribution suggesting agent-benchmark scores used to advertise commercial capability may carry unquantified contamination risk, which over 12 to 24 months could matter only if procurement or investment decisions cite agentic leaderboards without audit.
Technical Capability: Benchmark developers and eval consumers get an auditable framework showing that HAL-style leaderboards are broadly susceptible to contamination but that confirmed incidents are narrower than susceptibility, so headline agent scores should be read with provenance evidence attached.
Market & Valuation: Investors pricing agent startups on leaderboard rank should discount contaminated-benchmark scores until audits of this kind exist, since the paper shows even a 10-point benchmark-associated gap cannot be attributed to memorization with statistical confidence.