Jiaqiang Li, Yajie Yang, Zhiheng Xi, Jiadong Chen, Enyu Zhou, Senjie Jin, Yang Nan, Jiazheng Zhang, Han Wang, Yanxin Li, Dingwei Zhu, Bicheng Deng +6
Frontier multimodal agents often reach correct scientific conclusions without recovering the complete supporting evidence: answer accuracy exceeds complete-evidence recovery by more than 20 points, driven mainly by failures to assemble complete multi-panel evidence rather than by reasoning about evidence once found.
A new benchmark, Sci-MMR: 235 multi-hop reasoning tasks built from argument graphs over 800 peer-reviewed papers (Nature, Science, Cell, PNAS), spanning 4 disciplines and 35 domains, averaging 9 figure panels and ~10 hops per task. Eight frontier multimodal models were run under four settings (Caption-only, Direct Visual Reasoning, Evidence-Hint, Agentic Tool-Use), scored on answer accuracy, evidence coverage, and claim coverage, with human expert verification of all tasks and controlled interventions (cropping tools vs gold evidence).
Overall answer accuracy vs evidence-coverage gap exceeds 20 points (e.g., GPT-5.5 direct: 67.7% accurate but 58.5% evidence coverage, and complete-evidence recovery is much rarer: models locate at least one relevant region in 62.1% of runs but recover the full evidence set in only 12.7%). Error attribution under direct reasoning: 57.2% acquisition failures, 31.8% integration failures. Cropping tools add only +4.5 points, while gold evidence gains 19.1 to 37.0 points; even with gold evidence the strongest model reaches only 69.1% on hard tasks (overall hard-task accuracy is 10.6% direct, 17.0% agentic on average).
The authors concede in their Limitations section (full text truncated, so details are partial). Obvious unstated ones: only 235 tasks, a small and hard-skewed sample dominated by 2026 papers; evidence coverage and claim coverage are judged by LLM judges (they audit against human/cross-judge checks, but judge mismeasurement is a risk); the abstract's 'more than 20%' gap is averaged over very heterogeneous difficulty strata where hard-task scores sit near floor (4.9-29.6%), making averages noisy; single-run evaluations appear to be the norm, so variance is unquantified in what is shown; and the benchmark is LLM-generated then human-verified, so task phrasing and gold answers inherit pipeline biases.
A skeptic would note the headline gap partly reflects metric construction: demanding complete recovery of all gold evidence regions from 9-panel figures guarantees low coverage scores that answer-accuracy comparison overstates, and 'correct answer, incomplete evidence' may partly reflect models compressing rather than failing. The n=235 benchmark with LLM-as-judge scoring, one paper group authoring it, and no competitor baselines on the same construction means contamination-free difficulty is asserted rather than demonstrated. Gold-evidence interventions also hand the model the answer scaffolding, so the 37-point delta may conflate hint value with information leak.
Little or nothing yet for the AI economy in 12 to 24 months. The paper matters as an evaluation-methods signal: if agent-grade work products must be evidence-grounded (scientific, regulatory, medical), answer-centric benchmark gains may overstate deployable reliability, which tempers but does not directly measure any revenue or labor claim.
Technical Capability: Frontier-model buyers and builders get evidence that agentic multimodal reliability on long, multi-evidence tasks is far below headline benchmark accuracy, with a concrete diagnostic (acquisition vs integration) for where agents break.
Societal & Cultural: Anyone relying on autonomous research agents for scientific or evidence-based claims should care that models can assert correct conclusions without traceable supporting evidence, a hallucination-adjacent failure mode invisible to answer-only evals.