$τ$-Emotion-Bench: Exposing the Interaction Fragility of LLM Agents through Real-World User Emotions

Frontier LLM agents lose substantial task-completion reliability when the simulated user expresses emotions, and fine-tuning small models on interaction-aware trajectories (Tracer) narrows much of the gap.

A new 580-task, ten-domain tool-calling benchmark where a user simulator is conditioned on 594 emotion/behavior combinations; frontier and open-weight agents were evaluated with two simulators over three runs, plus Tracer fine-tuning of Qwen3 models (1.7B to 8B) on 1,958 generated trajectories.

With GLM-5.2 as simulator, best model GLM-5.3 gets 78.6% Avg@3 but only 66.5% Pass^3; with Qwen3-235B simulator, best is DeepSeek-V4.1-Flash at 55.9% Avg@3 and 32.2% Pass^3, so the consistent-completion gap is 12 to 24 points. Tracer-High lifts Qwen3 base models 17.1 to 25.2 points Avg@3 under GLM-5.2, with Tracer-High-8B reaching 54.7%; a low-data variant trained on only 1,498 trajectories from four domains generalizes to six unseen domains. All numbers are from the authors' own synthetic harness with simulated users, not human interactions.

The authors concede 101 of 580 tasks required manual correction, rely on model-based adjudication when deterministic checks are inconclusive, and the effect depends entirely on the quality of LLM-based user simulators, which they check for stability but cannot prove faithful. Unstated: emotions are imposed as structured conditions rather than arising naturally, human validation is limited to selected tasks, and Pass^3 on 50-to-70 tasks per domain has wide variance; no comparison to how much of the gap is emotion versus generic user-simulator variance.

The headline 'fragility' is measured against a second LLM role-playing emotion, so it may partly measure simulator style mismatch rather than real user-behavior fragility; results may be sensitive to the simulator prompt and the chosen emotion taxonomy; the benchmark's own construction and training pipeline come from the same authors, so the baseline gap that Tracer then closes is partially self-defined.

Little or nothing yet for AI economics in the next 12 to 24 months; this is an academic reliability benchmark and training-recipe paper. At most it is one more datum that agent reliability on multi-turn user-facing work is well below headline benchmark numbers, which matters only if emotion-conditioned variance reflects real customer-service deployment friction.

Technical Capability: Directly useful for teams building user-facing agents: Pass^3-style consistency metrics show average success overstates deployable reliability, and interaction-aware reward shaping is a concrete training lever.

Labor & Knowledge Work: If customer-facing automation depends on tolerating frustrated users, reliability numbers like these (about a third consistent completion under one simulator) temper near-term displacement claims for support roles.