The Unreliable Progress Bar: Can LLM Agents Reliably Report Task Progress Throughout Execution?

2609.08589 ↗ · 2026-09-08 · cs.SE, cs.AI, cs.CL

Boyang Wang, Yunhan Wang, Yalun Wu

33 pages, 13 figures, 20 tables

The finding

Headline claim

LLM agents cannot reliably report task-progress stage across a task's lifecycle: most deployed models are accurate at some stages and fail at others, with large mid-task accuracy drops or premature completion claims, so agent frameworks should not trust model-emitted progress signals to control continue-or-stop decisions.

The test

Two settings: (1) τ²-bench telecom with an added per-reply lifecycle-stage tag scored against environment state, across eleven to twelve deployed model configurations, and (2) StageIF, a controlled testbed of 12 scripted scenarios with five checkpoints each (four duty-active, one duty-inactive), 20 repetitions, scored at four failure layers (acting instead of reporting, omission, malformed report, wrong value), with seven of nine planned deployments passing identity checks and contributing 50,400 checkpoint positions.

Effect size & scope

Failures are large and stage-dependent: on τ²-bench, gpt-4.1 and gpt-5.5 are 90.6–99.4% correct before acting but only 5.8–11.5% mid-task; among five deployments with a mid-task trough, 82–90% of well-formed mid-task errors name an already-passed stage; some deployments (gemini-3.8-flash, envelope-stripped gpt-6-astra, claude-opus-5) stay 74–100% mid-task but fall to 48–64% at completion, always naming earlier states. On StageIF baseline arm, adherence at action-bearing checkpoints collapses (e.g., qwen3.5-35b-a3b: 21.7% wait, 1.2% ready, 2.9% step-done vs 97.9% done; Δ −89.3, 95% CI −93.8 to −83.9), while gpt-5.5's Δ of −4.2 is the only CI including zero. Four of eleven full-split deployments omit the required tag on 89.9–99.4% of checkpoints. Simple interventions (value, reminder, stage-information, reasoning changes) do not generally close the gap.

Limitations

The authors concede the benchmark's environment state cannot separate NEEDS_INFO from READY, position effects on τ²-bench are descriptive not causal, two deployments' strict-parser failures are conflated with JSON wrapping (stripped values are a diagnostic only), claude-opus-5 ran a partial split with simulator exclusions, and no claim is made about internal progress representations. Unstated: StageIF uses 12 synthetic scripted scenarios with frozen assistant text, a small and possibly unrepresentative checkpoint universe; deployment names bundle model and serving configuration so model-level conclusions are fuzzy; and the added reporting duty is an artificial instruction that real frameworks may elicit differently.

Counterpoint

The task is arguably artificial: models are graded on obeying an idiosyncratic tagging protocol bolted onto benchmarks never designed for it, so low scores may measure instruction-following on an unusual output format rather than any real self-monitoring deficit. Twelve synthetic StageIF scenarios is a tiny task base, prompt phrasing sensitivity is demonstrated by the parser and stripped-envelope discrepancies, and some models score near 100% at some stages, so 'unreliable' may describe a few weak models as much as a general capability gap. No strong external baseline (e.g., runtime-side state tracking, which the authors themselves note exists) is benchmarked head-to-head on cost and accuracy.

Economy implication

In the next 12 to 24 months this says little or nothing about AI economics directly; it is evidence that unattended agentic deployments in production workflows will need external, runtime-owned state tracking and verification rather than trusting model self-reports, which adds engineering cost to agent products but does not by itself establish whether agent ROI materializes.

Who this matters for

Technical Capability: Agent-framework builders get concrete evidence that model-emitted stop and progress signals fail in stage-specific ways, so control flow must be derived from environment state rather than model claims.

Labor & Knowledge Work: Teams deploying agents to run multi-step knowledge workflows unattended should treat premature 'done' claims as a measured failure mode requiring human or programmatic checkpoints before trust scales.

Bears on (advisory · papers never write evidence)

[1.3] Agentic reliability (multi-step completion without human correction) is improving materially, not just peak single-shot.

[B1] Agentic reliability gains will translate into enterprise ROI.

Abstract

Recent large language models can emit task-progress signals that agent frameworks use to decide whether a task should continue or stop, yet whether a model can reliably report its task progress at every stage of a task, and where and how its reports fail, has not been studied systematically. We evaluate this ability on the public benchmark $τ^2$-bench and on StageIF, a controlled testbed in which reporting checkpoints are placed across the task's lifecycle. Both settings require reports at multiple task stages. We find that reporting reliability depends on the stage a task has reached, and that almost every deployed model we test is reliable at some stages and unreliable at others. Where reporting breaks down is not the same everywhere. Most deployed models lose accuracy once work is under way and recover once the task is done. The newest generation closes that mid-task drop and instead grows conservative at the finish line. Our study exposes a capability gap in task-progress reporting and provides an evaluation protocol that spans the whole course of task execution for this ability on which agent operation depends. The findings indicate that agent frameworks should not control task flow on the strength of the model's state reports alone.