Overview of the NLPCC 2026 Shared Task 11: Agent-Based Experiment Reproduction from Scientific Papers

2609.11117 ↗ · 2026-09-10 · cs.CL

Hanhua Hong, Yizhi Li, Luu Gia Huy, Jian Yang, Ming Zhou, Chenghua Lin

NLPCC Shared Task

The finding

Headline claim

Current LLM agents still fail to reliably reproduce published scientific experiments, with execution and result matching (not code writing) as the bottleneck, best system scoring only 49.64% overall on a new process-oriented benchmark.

The test

NLPCC 2026 Shared Task on AgentActionBench: 150 papers (120 ML from ICML/ICLR/ACL/NeurIPS, 30 AI4Science), over 10,000 rubric items, agents run in a standardized MCP-recorded environment; scores from 3 participating teams plus a GPT-5.4-with-Codex baseline, evaluated by GPT-4o-mini on importance-weighted rubric pass rates across five workflow stages.

Effect size & scope

Best team scored 49.64% overall (46.69% ML, 61.25% AI4Science); the frontier baseline (Codex-GPT-5.4) scored only 24.19%; one team scored 7.03%. Execution (2.84-18.04%) and result matching (0.39-17.24%) were far below paper observation and plan writing, showing the gap is in running and verifying code, not writing it. Scope: 4 systems, 2 L40S GPUs, papers in a narrow ML/AI4Science slice, rubric scoring by a small evaluator model.

Limitations

Authors concede GPT-4o-mini as evaluator may be unreliable, rubric evaluation cannot eliminate hallucinated judgments, and AI4Science coverage is preliminary. Unstated: only 3 teams participated, human annotation covers just 15 papers (10%), model-generated rubrics were validated only by correlation on that small subset (n=15) and distort stage weighting toward Code Implementation, and the overall numbers rest entirely on model-judged rubrics.

Counterpoint

With 3 teams and a homegrown rubric pipeline judged by a cheap model, the 'current agents are far from reliable' claim rests on weak evaluators and possibly unfair or noisy rubric items; a single strong-lab system under prompt-optimization pressure could score much higher; correlations of 0.93/0.88 on 15 papers do not establish rubric validity at the claimed 150-paper scale; the benchmark may reward process-theater (logging the right actions) over actual reproduction.

Economy implication

Little or nothing yet in direct economic terms. The relevant signal is directional: agents that can write plausible code but not run and verify it will not yet displace lab-scientist or ML-engineer reproduction work, so any near-term AI ROI claims in R&D automation remain speculative for at least the next 12 to 24 months.

Who this matters for

Technical Capability: Quantifies that the execution stage of agentic coding, not code generation, is the binding constraint, which is where capability work should focus.

Labor & Knowledge Work: Suggests research-engineer and scientist reproduction tasks are not yet automatable, tempering near-term claims about AI replacing experimental research labor.

Bears on (advisory · papers never write evidence)

[1.3] Agentic reliability (multi-step completion without human correction) is improving materially, not just peak single-shot.

Abstract

Reproducibility is essential to scientific progress, yet the growing volume and complexity of scientific publications make exhaustive manual verification increasingly impractical. Although recent advances in large language model (LLM) agents enable automated experiment reproduction, existing evaluations largely focus on final repositories and are typically limited to machine learning (ML). We introduce AgentActionBench, a process-oriented benchmark for evaluating agent-based experiment reproduction across ML and AI4Science domains. Our framework uses an MCP-based Action Recorder to capture agents' behaviour throughout the reproduction process and evaluates the resulting traces with paper-specific rubrics. AgentActionBench contains 150 papers, including 120 ML papers and 30 AI4Science papers. A human-annotated subset covering 10% of the benchmark provides validation data, while model-assisted augmentation expands the full benchmark to more than 10,000 rubric items. Experimental results show that current systems remain limited, with execution as the primary bottleneck. Meanwhile, the strong Pearson and Spearman correlations between model-generated and human-annotated rubrics validate the reliability of our scalable rubric-generation approach.