Zixiang Chen, Yuheng Lu, Zihao Cheng, Zeming Liu, Jizeng Bai, Ziye Huang, Zhiyin Lin, Zihan Li, Yuhang Guo, Yunhong Wang, Haifeng Wang
Accepted to EMNLP 2026 (Main Conference)
State-of-the-art open-source GUI agents that perform well on single-device tasks fail substantially on cross-device workflows spanning Android, Windows, and Ubuntu, meaning existing benchmarks overstate real-world agent readiness.
The authors built JarvisGUI, a Docker-virtualized benchmark with dynamically composed multi-step, cross-device tasks (a typed task-graph system that chains atomic tasks and transfers files/state between platforms), then evaluated several open-source GUI agents (cited: Bai et al. 2025a, Zhou et al. 2025, Ye et al. 2025, Gu et al. 2025) using a two-stage Planner-Grounder architecture, scoring success via deterministic final-state evaluators.
The paper reports a 'substantial disparity' between single-device proficiency and cross-device competence, with failures attributed to fragile critical paths, cognitive/visual overload, dependency management, and platform bias in atomic actions. The provided text excerpt does not include the actual success-rate numbers, so the magnitude cannot be stated from the given material; the gap holds for the open-source agents tested in this virtual environment and may not extend to proprietary frontier agents not evaluated.
The authors concede prior coverage was thin (CRAB has only 18 cross-device tasks) and that their agents must be adapted to a single-platform-per-step paradigm via a Planner-Grounder wrapper. Unstated limitations: task graphs are largely auto-generated (sampling plus Qwen3-32B rewriting plus LLM-as-a-judge filtering), so realism depends on that pipeline; only open-source agents are tested, leaving frontier closed models unexamined; tasks run in virtualized, not real, device environments; results tables are not in the provided excerpt.
A skeptic would note the benchmark is partly synthetic (auto-composed task graphs, LLM-rewritten instructions, LLM-judge filtering of only 2,000 sampled tasks), the evaluated agents may be under-tuned for this distribution, and the Planner-Grounder adaptation itself could handicap agents; without closed-model baselines and without the actual numbers visible, the 'critical gap' could partly reflect a mismatch between agent training distribution and this new benchmark rather than a genuine capability ceiling.
Little or nothing yet for the economics of AI in the next 12 to 24 months: this is an evaluation-methods paper showing agents are further from unattended real-world workflow automation than single-device benchmarks suggest, which weakly cautions against assuming agentic reliability gains translate quickly into enterprise automation savings.
Technical Capability: GUI-agent builders get a new evaluation axis, cross-device state transfer and dependency management, that current training pipelines evidently do not optimize for.
Labor & Knowledge Work: If agents cannot reliably chain work across devices and platforms, the near-term automation threat to multi-application knowledge workflows is more distant than benchmark scores imply.