ToolBottleneckBench: Measuring Tool-Agent Performance under Controlled Pressures and Fixed Harness Budgets

Apparent tool-agent failures on enterprise-style tasks are mostly artifacts of harness budgets, especially completion-cap truncation under large visible tool menus, rather than genuine model capability deficits.

A controlled benchmark (36 synthetic task families, four assigned pressures varied one at a time) run across 21 model-and-harness configurations from 7 organizations, 26,576 scored conditions (20,408 exact-graded), with an exact nine-obligation grader recording first unmet obligation plus terminal harness-limit flag.

Only the visible-menu pressure has a large penalty: growing the menu from 8 to 128 tools costs 12.3 points of verified task success under a routed menu and 17.5 under a fixed menu. Menu penalty tracks truncation at the 1,024-token completion cap (Pearson r = -0.95); non-truncating configurations stay within 2.8 points of zero; a sixteenfold cap raise cuts the penalty from -43.7 to -2.7 points in truncation-heavy configurations. Workflow pressure has no detectable pooled effect, encoding effects are small. 91.5% of graded failures end at a harness limit, only 5.5% are wrong conclusions. Scope: synthetic enterprise tasks, managed inference endpoints, one frozen harness contract.

Synthetic tasks with opaque IDs may not reflect real enterprise environments; the routed menu is oracle-assisted (uses ground truth to narrow); workflow is a composite pressure making interpretation messy; failure labels locate but do not explain failures; the authors concede decoders identify the assigned pressure from the failure record no better than a simple tool-call count; prompt repairs did not help; only two repairs beat configuration switching and neither improves model behavior. The truncation finding depends on a deliberately tight 1,024-token cap that may be tighter than typical production settings.

The headline result may be partly manufactured by the experiment design: a 1,024-token completion cap plus large tool schemas (~55k bytes at V=128) guarantees truncation, so finding that truncation explains the penalty is close to circular. All tasks are synthetic and graded exactly, so nothing here shows real-world agent failures are harness artifacts; a skeptic would note the authors' own grader-derived failure labels carry no diagnostic information beyond a call count, and that vendor benchmark gaming is precisely what a fixed-budget benchmark like this is meant to police but cannot alone prove.

Little or nothing yet for direct economics. The indirect implication is that reported agent-benchmark gaps between vendors may partly reflect harness and serving-configuration choices rather than model quality, which would compress apparent differentiation among frontier models and complicate procurement decisions, but this paper measures only synthetic tasks under its own budgets.

Technical Capability: Benchmark and harness builders should report pressure effects together with the budgets under which they were measured, since 91.5% of failures ended at a harness limit rather than a wrong answer.

Market & Valuation: Enterprise buyers comparing vendor agent scores should ask about completion caps, menus, and serving adapters before paying premiums for headline benchmark gaps that may be harness artifacts.