Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents

2607.28227 ↗ · 2026-07-30 · cs.AI, cs.CV · h-max 5 · 0 citations

Hanzhang Zhou, Panrong Tong, Xu Zhang, Quyu Kong, Chenglin Cai, Tianyu Xia, Gongjie Zhang, Jianan Zhang, Long Li, Long Chen, Lei Wang, Gaole Dai +4

The finding

Headline claim

A single foundation GUI agent trained with real-device data, hybrid GUI+CLI actions, and scaled online RL achieves state-of-the-art performance on mobile-use benchmarks and competitive performance on computer- and browser-use tasks against frontier closed-source models.

The test

Evaluation across mobile (MobileWorld, MobileWorld-Real with 400+ tasks on 100+ physical phones, AndroidDaily), computer use (OSWorld-Verified, OSWorld-v2), browser (WebArena, BrowseComp), and GUI grounding (ScreenSpot-Pro) benchmarks, against frontier models including Opus 4.8, Gemini 3.1 Pro, GPT-5.6 Sol, and Seed 2.1 Pro; training uses 100+ real devices, ~10,000 concurrent sandbox environments, and online RL on trajectories exceeding 100 turns.

Effect size & scope

On MobileWorld-Real the agent scores 92.2%, beating the best frontier baseline (Seed 2.1 Pro) by 3.5 points and Claude Opus 4.8 by 7.5 points; 82.1% on MobileWorld (a 14.6 point lead over Opus 4.8); 79.5% on OSWorld-Verified (second overall); 73.6% on WebArena; 81.5% on ScreenSpot-Pro. The advantage is largest on the authors' own real-device benchmark and on their home platform family, and narrows or reverses to second-place on harder third-party benchmarks like OSWorld-v2 (13.9% binary success).

Limitations

The authors concede the simulation-to-real gap and real-world scenario challenges (interruptions, permissions, varied layouts) remain, and list limitations in a section not included in the provided text. Unstated but obvious: MobileWorld-Real and AndroidDaily are the authors' own benchmarks, so the headline 92.2% and 97.5% numbers come with a direct conflict of interest; user takeover on high-risk actions and ask_user actions complicate what counts as unattended success; the provided text omits baseline evaluation protocols and whether frontier competitors were tuned for these benchmarks; the report is a vendor artifact with no independent replication.

Counterpoint

A fair skeptic would note that a lab evaluating itself on a benchmark it designed, on the platform family its own model was trained on, with an in-house AutoJudge trajectory-level judge, is a textbook setup for favorable results; the third-party benchmark results (second place on OSWorld-Verified, 13.9% on OSWorld-v2) suggest the mobile numbers are where selection effects are strongest; the frontier baselines cited are named inconsistently in the text (Gemini 3.1 Pro vs 3.5 Flash, GPT-5.6 Sol vs GPT-5.5), and no code or eval harness details are verifiable from the provided text.

Economy implication

Little or nothing yet for economics directly: this is a capability report with no cost, pricing, or deployment data. If the real-device numbers hold under independent evaluation, they would support the plausibility that mobile GUI agents are approaching deployment-grade reliability, which is a precondition (not evidence) for agentic enterprise ROI; that bridge remains untested here.

Who this matters for

Technical Capability: The report is direct evidence that GUI-agent progress is shifting from simulated benchmark optimization to real-device training and evaluation, with hybrid GUI+CLI action spaces and batched actions as concrete architectural moves.

Market & Valuation: Alibaba shipping a competitive cross-platform agent matters for the competitive frontier-clustering story: another non-US lab claims parity or leads over Anthropic, OpenAI, and Google on agent benchmarks, with no cost or latency data to judge whether the lead is economically usable.

Bears on (advisory · papers never write evidence)

[1.3] Agentic reliability (multi-step completion without human correction) is improving materially, not just peak single-shot.

[5.1] Frontier models cluster tightly (small gaps, fast catch-up).

[6.1] Chinese frontier models lag US frontier by a widening margin.

Abstract

GUI agents have the potential to become a general purpose executor over existing digital devices. To advance them toward real-world use, we envision agents that operate reliably on real devices, execute workflows across platforms, combine GUI interaction with CLI execution, complete long-horizon tasks, proactively initiate useful services, and autonomously improve their capabilities with minimal human effort. Guided by this vision, we present Qwen-UI-Agent, a real-world centric foundation GUI agent spanning mobile, computer-use, web, and DeepSearch environments. Qwen-UI-Agent combines diverse sandbox environments with a large-scale real-device mobile runtime. Its unified action space interleaves GUI operations with CLI execution and generates batched actions in a single model turn. An AutoResearch-style data flywheel uses agents to construct tasks and environments, diagnose failures, and plan subsequent iterations. Online RL supports training on trajectories exceeding 100 turns, with over 10,000 concurrent environments accelerating rollout. A lightweight harness layer supports proactive service initiation and stateful workflows across mobile and computer. Across a broad suite of evaluations, Qwen-UI-Agent sets state-of-the-art performance on mobile-use benchmarks while delivering competitive performance on computer- and browser-use tasks against frontier models, including Opus 4.8, Gemini 3.1 Pro, and GPT-5.6 Sol. On mobile use, it achieves 82.1% on MobileWorld, 92.2% on MobileWorld-Real, and 97.5% on AndroidDaily. On computer use, it achieves 79.5% on OSWorld-Verified and a 40.0% partial-progress score on OSWorld-v2. On browser use and GUI grounding, it achieves 73.6% on WebArena and 81.5% on ScreenSpot-Pro, respectively.