$τ^τ$-Bench: An Environment for End-To-End, Realistic Agent Construction

2609.04611 ↗ · 2026-09-04 · cs.AI

Quan Shi, Keshav Dhandhania, Karthik Narasimhan, Victor Barres

41 pages, 13 figures, 6 tables

The finding

Headline claim

When coding agents are asked to build a complete customer-service agent end to end from realistic business records, an interactive client, and cost constraints, the best system delivers only 23.9 percent of held-out evaluation simulations, far below an 82.2 percent expert ceiling.

The test

Sierra and Princeton built tau-tau-bench: 53 tasks across four domains where a developer agent must recover specifications from 2,868 generated business artifacts (over 5.5 million text tokens plus images and recordings), interview a simulated client, integrate a possibly defective client REST API, inherit buggy codebases, and serve within a per-conversation credit budget on a fixed model menu; submissions are scored by deployment against held-out tau-bench-style simulated users.

Effect size & scope

The strongest configuration measured, Claude Opus 5 under Claude Code, passes 23.9 percent of evaluation simulations versus an 82.2 percent expert-authored reference, so the best AI system reaches roughly 29 percent of the human ceiling. Failures are qualitative too: shallow keyword queries instead of reading the corpus, almost no client interviewing, little architectural exploration, mismanaged budgets, and self-authored tests that mirror their own blind spots. Results are limited to customer-service agent construction in four tau-bench domains under this specific harness.

Limitations

The authors concede their domains are public tau-bench policies, so training-data contamination is a live risk and they author alternative versions to mitigate it; artifacts are model-generated and only human-audited, the client and user are simulated, one live-experiment call is allowed, and the private held-out set is needed to fully verify contamination resistance. Unstated: the expert ceiling is a single authored reference, task count per domain is small, and all numbers are simulated conversation metrics rather than real production deployments.

Counterpoint

This is a lab benchmark from Sierra, a company selling agent tooling, whose design choices (LLM-simulated clients and users, judge-graded messaging rubrics, generated evidence) could systematically penalize unfamiliar agent designs; scores may be prompt- and harness-sensitive, and a 23.9 percent pass rate under a deliberately adversarial end-to-end setup does not mean agents fail at narrower, well-specified construction tasks where current tools already shine.

Economy implication

Little direct signal for near-term economics, but it argues that fully autonomous agent construction, the labor-intensive middle of AI consulting and delivery, remains far from solved in the next 12 to 24 months, and that human developers still own requirements gathering and validation even when AI writes most of the code.

Who this matters for

Technical Capability: Provides a measurable, harder frontier for coding agents beyond SWE-bench-style issue fixing: specification recovery, stakeholder elicitation, and cost-constrained design are now scored.

Labor & Knowledge Work: If agent construction is the bottleneck AI cannot yet do end to end, the near-term job of AI engineers shifts toward the specification and validation work this benchmark shows models lack, rather than pure coding.

Market & Valuation: Benchmark results from an agent-platform vendor should be read partly as market positioning for the difficulty of the problems Sierra's product addresses, and as evidence that autonomous enterprise agent delivery is not yet commoditized.

Bears on (advisory · papers never write evidence)

[1.3] Agentic reliability (multi-step completion without human correction) is improving materially, not just peak single-shot.

Abstract

LLM agents are rapidly becoming production software, deployed to handle customer service, adjudicate disputes, and operate internal systems. Notably, the work of building them is increasingly handed to coding agents, yet existing benchmarks say little about whether an AI system can deliver one under the conditions of a real client engagement. We introduce $τ^τ$-bench (pronounced hyper-tau-bench), a benchmark that makes agent construction the task. A developer agent is given the records a business actually keeps, a client who holds requirements, a production API that operations must run through, a codebase to inherit, and limits on serving cost and models: the same starting point a real engagement provides. From these it must deliver a complete customer-service agent, scored by deploying that agent against held-out simulated users. Across 53 tasks spanning four domains, the strongest configuration, Claude Opus 5 under Claude Code, passes just 23.9% of evaluation simulations. Meanwhile, an expert-authored reference ceiling scores 82.2%. The failures mirror ones human agent developers see: models issue shallow queries in place of deep comprehension of the records, communicate almost nothing to the client, and experiment too little with agent architecture and serving spend, shipping the first design that runs. We aim for $τ^τ$-bench to turn the work of cooperative agent building into a measurable target for coding agents.