Calibration is the Bottleneck: An Action-Class Diagnostic of Multi-Turn Tool-Calling

2609.00949 ↗ · 2026-09-01 · cs.CL, cs.AI

Kangjia Zhao, Jiajun Li, Haozhan Shen, Wei Chow, Linfeng Li, Hang Song, Lingdong Kong, Chen Zhi, Tiancheng Zhao, Songhua Liu, Jianwei Yin

Accepted to Findings of EMNLP 2026. Code: https://github.com/fbj2333/tool-calling-calibration

The finding

Headline claim

Aggregate tool-calling accuracy conceals a large, family-specific failure mode: models emit the wrong action class (calling a tool when the context requires asking or refusing), and the BFCL state grader masks these misses, so leading open-weight models' benchmark standing is partly inflated.

The test

A four-class action-space diagnostic (tool_call/ask/refuse/confirm) with Gold Action Recall versus BFCL v3 multi-turn conversation accuracy (200 conversations per category), run on 22 open-weight checkpoints across 11 families plus 4 closed API anchors, cross-validated on tau2-bench retail and airline, with a 200-case human audit (kappa 0.92) and two inference-time probes (SRI, CRI) testing calibration plasticity.

Effect size & scope

xLAM-2-8b has 71.8% overall accuracy but only 19.5% GAR on miss_param, with state-grader masking accounting for up to 66 pp of apparent accuracy (xLAM-2-70b miss_param Delta of -66.0); a single SRI perturbation moved accuracy in opposite directions across families by up to +11.5 vs -21.0 pp; scaling within Qwen3 moved miss_param GAR only 5.5 pp while Acc rose 14.5 pp.

Limitations

Authors concede the diagnostic is train-free and prescribes no fix, covers only four benchmark categories and four action classes with fixed gold labels defined by benchmark design, and the intervention probes are diagnostic rather than production solutions; cue-based classification of ask/refuse/confirm from text is inherently noisy, and findings rest mainly on BFCL v3 with only partial replication on tau2-bench (50 airline tasks).

Counterpoint

The gold action classes are constructed by the benchmark authors' own perturbation scheme, so calling the observed behavior miscalibration partly assumes the benchmark's conception of correct behavior; narrow tool-trained models inflated on state grading could reflect training distribution mismatch with synthetic perturbations rather than a deployment-relevant flaw, and cue-list labeling plus a single benchmark family limits generalization.

Economy implication

Little or nothing directly yet, but if the masking result generalizes, reported agent benchmark gains overstate readiness for unattended consequential workflows, which matters for how quickly enterprise agent ROI claims convert to deployed reliability in the next 12 to 24 months.

Who this matters for

Technical Capability: Benchmark builders and agent developers learn that state-graded trajectory accuracy systematically hides action-class failures, and that GAR should be reported alongside Acc.

Market & Valuation: Buyers weighting vendor claims off public tool-calling leaderboards should discount aggregate scores for tool-specialized models until per-scenario diagnostics are available.

Bears on (advisory · papers never write evidence)

[1.2] Benchmark gains are increasingly contaminated / saturated, overstating progress.

Abstract

Multi-turn tool calling is a core evaluation scenario for large language model (LLM) agents. On public tool-calling benchmarks, open-weight models now approach or even surpass closed-source frontier models in aggregate accuracy. However, this metric averages over many different multi-turn situations and obscures whether progress is balanced across them. We propose an action-class-oriented diagnostic framework that decomposes multi-turn failures into two orthogonal modes: action-class miscalibration and action-execution failure. The framework operates over a four-class action space (TOOL_CALL/ASK/REFUSE/CONFIRM) and introduces a self-revealing upper bound Acc <= GAR (Gold Action Recall); the two modes show up as bound violation (Acc > GAR, exposing state-grader masking of miscalibration) and large bound slack (GAR >> Acc, localizing execution failure within TOOL_CALL). We validate it on a panel of tool-calling models across multiple multi-turn benchmarks. Across our panel, the diagnostic reveals action-class miscalibration as a substantial failure mode the state grader cannot see. This gap inflates standing for heavily tool-trained families, which our diagnostic separates from families with context-appropriate action choice. Calibration is reshapable through context-only perturbations, but the reshape is heterogeneous: a single perturbation moves accuracy in opposite directions across families (up to +11.5 vs -21.0 pp on the same scenario), and its effect further depends on the perturbation mechanism. We argue that multi-turn tool-calling evaluations should supplement aggregate accuracy with action-class diagnostics that expose what the model actually does in each scenario.