Mr.LHDR: A Benchmark for Multimodal Real-World Long-Horizon Deep Research Agents

2609.11318 ↗ · 2026-09-10 · cs.AI

Minghao Guo, Meng Cao, Sui Zhao, Siyu Ning, Xin Wang, Haoze Zhao, Jiaxuan Yang, Haihong Hao, Mingfei Han, Shunlin Rong, Haijun Wu, Xiaodan Liang +1

Code and data are available at https://github.com/minghaoguo20/Mr-LHDR-eval

The finding

Headline claim

On long, dependency-chained multimodal research tasks, current frontier and deep-research agents are far from reliable: the best model achieves only 43.1% overall and 34.3% strict accuracy, and final-answer correctness substantially overstates complete, dependency-consistent task success.

The test

A new benchmark, Mr.LHDR: 102 hand-annotated open-web research questions across eight categories, each with a hidden dependency DAG averaging 12.1 necessary intermediate conclusions and depth 10.4, requiring at least one non-text evidence step. 25 systems (regular models, dedicated deep research systems, framework agents) were scored with OA, SA, CS, and a new prerequisite-gated DACS, with LLM-judge grading, bootstrap CIs, cross-judge agreement checks, an image ablation, and checklist-length analysis.

Effect size & scope

Best regular model GPT-5.5: 43.1% OA, 34.3% SA; best dedicated system o3 Deep Research: 32.4% OA, 19.6% SA; DeerFlow (qwen3-vl-235b): 15.7% OA, 9.8% SA. Image ablation: removing images drops DACS from 34.2% to 21.6% (12.6 points, 95% CI [6.9, 18.9]). SA decreases monotonically with checklist length for all representative models. Authors concede confidence intervals overlap for many neighboring systems, so no fine-grained ranking is supported.

Limitations

Authors concede: only 102 questions (narrow coverage), judge-based grading rather than observing internal reasoning or actual retrieval, metrics score stated conclusions not true evidence-chain completion, overlapping CIs prevent separable rankings, AI-assisted construction pipeline may leak construction patterns, and DACS depends on the correctness of the annotated dependency DAG itself. Unstated: possible contamination if models trained on the released data, prompt sensitivity of the LLM judge, and questions may be harder than typical economic research tasks by design.

Counterpoint

A fair skeptic would note this is a benchmark paper with an n=102 test set and an author-defined notion of 'necessary' intermediate conclusions; the DAG may encode one canonical path among several valid research routes, penalizing agents that reach correct answers via different (unstated) chains. The DACS gap between models and dedicated systems partly reflects prompt/format behavior, and LLM-judge scores on long free-form responses are known to be noisy. Benchmark-hardness-by-construction says little about real-world task distribution.

Economy implication

Little or nothing yet for AI economics in the next 12 to 24 months directly; the finding that dedicated deep-research products complete under 20% of long dependency chains strictly suggests agent reliability claims should be discounted in near-term enterprise ROI projections for research-heavy workflows, but at benchmark scale this is a caution, not evidence.

Who this matters for

Technical Capability: Researchers and builders of deep-research agents get the first dependency-gated metric showing that final-answer accuracy masks roughly 9 to 13 points of missing or unsupported reasoning state.

Market & Valuation: Investors pricing agentic 'deep research' products should note the strongest dedicated system strictly completes under 20% of long chains, tempering near-term automation narratives for research workflows.

Bears on (advisory · papers never write evidence)

[1.2] Benchmark gains are increasingly contaminated / saturated, overstating progress.

[1.3] Agentic reliability (multi-step completion without human correction) is improving materially, not just peak single-shot.

Abstract

Deep research agents are increasingly capable of web search, tool use, multimodal evidence analysis, and information synthesis. However, existing benchmarks mainly evaluate medium-horizon exploration and rarely test whether agents can sustain long, dependency-heavy research processes. We introduce Mr.LHDR (Multimodal real-world Long-Horizon Deep Research), a benchmark for evaluating real-world deep research over long, irreducible chains of interdependent evidence across eight categories. Each question is constructed from a hidden Node-Relation graph and requires an average of 12.1 necessary intermediate conclusions with a mean dependency depth of 10.4 before reaching a short, unique, and verifiable answer. Questions incorporate multimodal evidence, including images, maps, PDFs, logos, charts, tables, and video frames, with at least one non-text element that changes the reasoning state. Mr.LHDR evaluates both final answers and the correctness of intermediate conclusions under annotated dependencies. We evaluate general models, deep research systems, and agent frameworks using Overall Accuracy (OA), Strict Accuracy (SA), Checklist Score (CS), and Dependency-Aware Checklist Score (DACS). Results show that even the strongest system achieves only 43.1% OA and 34.3% SA, indicating that final-answer accuracy substantially overestimates complete research success. Removing images reduces DACS by 12.6 points, demonstrating the importance of multimodal evidence, while SA consistently declines as reasoning chains become longer. These findings reveal sustained, dependency-consistent evidence integration, rather than isolated fact retrieval, as a key bottleneck for current deep research agents.