An empirical study from Surge AI evaluated 12 frontier models (including GPT-5.2, Claude Opus 4.5, Gemini 3 Pro, and others) on 150 multi-step customer-support tasks in a simulated e-commerce RL environment, finding that even the best-performing model (GPT-5.2) fails approximately 40% of tasks. Failure analysis reveals a consistent five-level capability hierarchy, tool use, planning, adaptability, groundedness, and common-sense reasoning, with weaker models failing at levels 1–2 and stronger models bottlenecked at levels 4–5. The 40% failure rate on domain-realistic autonomous tasks, even at the frontier, provides a concrete reliability baseline for financial institutions evaluating deployment risk in customer-facing or workflow-automation use cases.
335 other tracked developments touching the same claims.