Jianlyu Chen, Yuyang Hu, Hongjin Qian, Jiawei Liu, Wenqing Wei, Xiaolong Chen, Defu Lian, Zhicheng Dou, Chaozhuo Li, Qiwei Ye, Zheng Liu
48 pages, 3 figures
Automatically distilling GitHub repositories and papers into verified skill libraries, and loading them as operating context, materially improves a frontier coding agent's performance on autonomous ML research benchmarks with model, harness, and compute budget held fixed.
The DisCo system built the AREX-Skill Library (5,000+ verified skills from 1,000 widely used ML repositories, routed over 20 areas and 178 capability families), then compared the same GPT-5.5-backed research agent with and without skills on four agentic benchmarks: MLE-bench, PaperBench, FrontierCS (agent track), and PassNet, holding backbone, harness, and execution budget fixed; skill construction was completed before downstream runs.
Relative to the no-skills agent, scores rise 134.3% on MLE-bench, 34.4% on PaperBench, 9.2% on FrontierCS, and 14.0% on PassNet; gains are largest on low-baseline and harder tasks, a small number of PaperBench tasks regress, and skills improve the score-efficiency frontier beyond raw resource scaling on FrontierCS.
The authors concede a small number of task regressions and that some skills are descriptive rather than executable with recorded gaps. Unstated: several benchmark-specific skill pools (PaperBench, MLE-bench, FrontierCS, PassNet) were constructed task-oriented, raising leakage-adjacent concerns; verification quality is self-defined; the 1,000-repository snapshot may not generalize beyond ML research; no cost accounting for the skill-construction compute itself.
Benchmark gaming is the obvious worry: task-oriented skills distilled with knowledge of the target benchmark, including recovery knowledge and paired verification trials, blur the line between pre-task knowledge and test-specific tuning; the 134.3% headline comes from the benchmark where such construction is easiest; no ablations are summarized for how much of the gain is generic operational knowledge versus benchmark-adjacent hints; small regressions suggest fragility rather than a stable knowledge layer.
If distilled skill libraries transfer to real enterprise ML and software workflows, they offer a cheap, model-agnostic way to raise agent output without paying for frontier model upgrades, which would push value toward whoever owns curated operational-knowledge assets and slightly erode the premium on raw model capability. At this stage the evidence is confined to research benchmarks, so the honest 12 to 24 month read is a signal about where differentiation may settle, not demonstrated enterprise ROI.
Technical Capability: Provides direct evidence that a third layer, curated operational knowledge between model and harness, can lift agentic performance without retraining or scaling compute.
Labor & Knowledge Work: If skills cheaply substitute for expert know-how in ML engineering tasks, the scarce expertise premium that ML engineers command could compress.
Market & Valuation: Suggests possible value migration toward proprietary skill and knowledge-layer assets over raw model access, relevant for where rents in the agent stack settle.