Kolab: Benchmarking Human-Agent Collaboration in Long-Horizon Workflows

2610.21109 ↗ · 2026-10-15 · cs.AI, cs.CL, cs.HC, cs.LG · h-max 4 · 0 citations

Shirley Anugrah Hayati, Patricia Anugrah Setiani, Ruizi Wang, Zhouyi Li, Khanh Chi Le, Young-Jun Lee, Dongyeop Kang

The finding

Headline claim

Frontier multimodal models cannot reliably anticipate how human experts will use their outputs (adopt, modify, reject) or judge collaboration quality from real professional workflows, and more context or modalities does not consistently help.

The test

Kolab dataset: 115+ hours of screen recordings from 50 professionals across 40 occupations solving self-selected long-horizon problems with agents (mainly Claude), yielding 692 annotated adoption instances (Adopt 65.9%, Modify 19.1%, Reject 15.0%) across 100 workflows and 508 steps. Frontier MLLMs and agents were benchmarked on forecasting adoption behavior and predicting human collaboration-satisfaction and output-quality ratings, with varied context lengths and modalities.

Effect size & scope

Gemini 3.1 Pro reaches 39.1% F1 and GPT-6 51.3% F1 on adoption forecasting; collaboration-satisfaction predictions correlate only weakly with expert ratings (Spearman rho = .08 and .38). Models show systematic biases: favoring certain adoption labels and overpredicting high ratings. Longer workflow context does not consistently improve predictions. Scope: one-hour professional sessions with self-chosen tasks, paid $100 participants, mostly collaborative and positively rated sessions.

Limitations

Authors concede models overestimate collaboration quality and rely on prediction tendencies rather than understanding workflows; ratings are heavily skewed positive (collaboration satisfaction 5 in 54.9% of sessions), which limits the discrimination task; graph annotation was partly refined by the research team; participants were recruited via Prolific/Upwork and may not represent professional populations; single primary agent (Claude) per session.

Counterpoint

The benchmark task of predicting adoption from screen recordings is far removed from what a deployed agent can observe at inference time, so low F1 may reflect an unfair or proxy task rather than genuine collaboration incapability; 50 participants and 692 instances is a small sample with annotation subjectivity in Adopt/Modify/Reject boundaries; rating skew plus prompt sensitivity could produce the weak correlations; results on a 2026-era model set may not extrapolate.

Economy implication

Little or nothing yet for near-term economics directly, but it complicates the assumption that strong agentic benchmark scores translate into productive human-AI collaboration: if agents cannot anticipate adoption or monitor collaboration quality, enterprise co-pilot workflows may need more human oversight than ROI models assume, slowing B1-style automation savings.

Who this matters for

Labor & Knowledge Work: Knowledge workers considering whether agents can slot into real multi-step workflows should note that agents misjudge when their contributions will be used, implying continued human coordination burden rather than replacement.

Technical Capability: Benchmark builders and agent developers get the first multimodal, multi-domain benchmark of actual human-agent collaboration trajectories, showing autonomous single-shot evals do not measure collaborative competence.

Bears on (advisory · papers never write evidence)

[1.3] Agentic reliability (multi-step completion without human correction) is improving materially, not just peak single-shot.

[B1] Agentic reliability gains will translate into enterprise ROI.

Abstract

When humans collaborate with AI agents on long-horizon tasks, what happens to the agents' contributions? For solving long-horizon tasks, humans place greater importance on ongoing interaction with AI agents (e.g., revising plans) than on an agent's ability to complete the task in a single attempt. However, most agentic benchmarks have focused more on automation. Strong performance on such benchmarks does not imply effective real-world collaboration. Professional workflows offer a useful setting, as they often involve interdependent steps. To address this gap, we introduce Kolab, a new dataset of 115+ hours of screen recordings of 50 professionals from 40 occupations collaborating with agents on realistic, long-horizon problems. Kolab also contains participants' collaboration and output quality judgments. Our analysis reveals diverse collaboration patterns, highlighting the importance of when and how professionals coordinate with agents. A collaborative agent must do two things that automation benchmarks never test: anticipate whether experts will adopt, modify, or reject its contribution once it leaves the chat window to guide its next steps, and assess collaboration quality from the unfolding workflow without constant expert feedback. Our benchmark operationalizes both: (1) forecasting how professionals use agent contributions at diverse workflows, and (2) verifying their judgments of collaboration satisfaction and output quality. Findings show that frontier models, both agentic and non-agentic, overestimate collaboration with experts and struggle to distinguish between high- and low-quality collaboration. Longer workflow context does not consistently improve these predictions. As frontier agents cannot yet understand long-horizon collaboration with experts, we position Kolab as a valuable dataset for studying human-agent collaboration grounded in real professional workflows.