Divyansh Singh
Controlled exposure of benchmark-specific information during fine-tuning causally raises executable-benchmark accuracy by +7.17 to +27.31 percentage points, showing benchmark scores can partly reflect memorized private associations rather than capability.
A randomized difference-in-differences design: 2,048 fresh synthetic four-choice task families (SQLite and deterministic Python domains) paired with private key-value associations, LoRA adaptation of two models (Qwen3-4B-Instruct-2507 and Gemma 4 31B) with exposure to 512 families' keys per domain versus balanced no-key controls, 262,144 scored generations with executable oracles certifying exactly one correct candidate.
Exposure-attributable DiD effects: Qwen SQL +7.17 pp [4.53, 9.78], Qwen Python +27.31 pp [22.86, 31.75], Gemma SQL +12.92 pp [8.72, 17.13], Gemma Python +10.14 pp [7.26, 13.18], mean +14.39 pp, all CIs exclude zero. Holds only for controlled post-training LoRA exposure in synthetic four-choice executable tasks on two open-weight models; authors explicitly do not claim this estimates web-scale contamination effects.
Authors concede: single adapter training realization per cell (no training-seed variation in CIs), only two model families and two domains, four-choice semantic decisions rendered by a trusted compiler rather than unrestricted generation, and effects do not extrapolate to naturally occurring contamination. Unstated: the private-association design measures memorization of an arbitrary key-fact mapping, which may overstate effects for benchmarks whose 'contaminated' content is reasoning steps models could partially derive; also AI tools were used pervasively, and the synthetic task distribution has no demonstrated relationship to real benchmark items.
The instrument measures exposure to a single arbitrary key-value fact explicitly inserted into the training mixture, which is a much cleaner and stronger signal than the paraphrased, diluted, and derivable leakage that occurs in real web-scale pretraining; the +27.31 pp Qwen Python cell also shows accuracy dropping in the control arm after adaptation, suggesting the DiD partly captures adaptation instability rather than pure memorization; two small-ish models and two synthetic domains with one training seed each is thin coverage for claims about benchmark interpretation generally.
Little or nothing yet for AI economics in the next 12 to 24 months. At most it reinforces, with causal rather than correlational evidence, the already-held view that static benchmark scores overstate frontier-model differentiation, which matters for procurement and valuation only insofar as buyers already discount headline benchmark numbers; it offers no new measurement of costs, adoption, or labor effects.
Technical Capability: Gives evaluation researchers a template for measuring contamination consequence experimentally rather than only detecting provenance, and shows the exposure effect can be as large as 27 points.
Market & Valuation: Supports discounting vendor-reported benchmark deltas when comparing frontier models, since a meaningful slice of such gains could reflect exposure rather than capability.
Regulatory & Legal: If regulators or procurement standards rely on benchmark scores to certify model capability, this paper shows those scores are causally sensitive to training-data access, strengthening the case for contamination disclosure requirements.