Executable Prefix Values for Locating Agent Failures

LLM-judge 'first mistake' labels on failed agent trajectories do not mark the point where a run actually becomes unrecoverable; measuring the agent's success probability at each step (the V-curve) does, and restarting at the measured point improves outcomes while restarting at judge labels does not.

On SWE-bench Verified, two frozen open-weight agents (Qwen3.6-27B via mini-swe-agent, plus a Qwen3.5-9B policy in the appendix) ran 150 tasks 8 times each (1,200 runs). For 47 failed trajectories (60 for the second policy), the authors reconstructed intermediate container states and ran 331 continuations on average per trajectory to estimate the success probability at each step, with a certified point of no return, then probed every step labeled by four frontier judges (GPT-6.1 Sol, GPT-5.5, o3, GPT-6 Astra) under the Traverse rubric, plus Claude Opus 5.5 and a human annotator on a 40-trajectory subset.

Cross-lineage judge agreement on the exact failing step is 21/57 for the most similar distinct pair and 3/57 for all three main judges; the two GPT-6 judges agree 52/57, so consensus is inflated by near-duplicate lineage. For 66% of labels the measured success probability does not drop at the labeled step; only 7/188 judge answers name the decisive step; GPT-6 judges flag 'mistakes' in 8/10 and 10/10 of successful decoy runs. Restarting at the measured restart point raises expected success from 0.44 to 0.57 offline and is confirmed online, whereas restarting at judge labels yields -0.08 to -0.03. Scope: only 21 of 150 tasks (14%) sit in the informative pass-rate band; results hold for one scaffold, two open-weight agents, and one benchmark.

The authors concede heavy selection: only 21 tasks fall in the band where failure localization is meaningful, 74% of selected trajectories were replayable, the isotonic fit flattens genuine recoveries, and the estimator's parameters were tuned before validation on real curves. Unstated limitations: very small n (47 and 60 trajectories), a single minimal scaffold whose results may not transfer to production agents, V is policy-relative so the 'cause' is defined only against this agent's own behavior, and the restart evaluation is partly offline against estimated curves.

A skeptic would note the comparison is partly self-serving: the V-curve is expensive (331 continuations per failure) and the paper's own restart policy uses that same expensive signal, so the practical question is whether a cheap estimator beats a cheap judge, not whether the gold standard beats the judge. Judge labels are also being scored against a rubric (earliest erroneous step) different from what V measures (point of value collapse), so 'labels scatter' may partly reflect different constructs, not judge incompetence. The 0.44-to-0.57 offline gain comes from a restart rule built on the same curves used to estimate it, risking optimism, and the online check is small. The 'band' restriction to 2-6 successes of 8 makes the pool tiny and may make V-curves unusually well-behaved.

Little or nothing yet for the economics of AI. The paper bears indirectly on the agentic-reliability-to-ROI bridge: if judge-labeled failure attribution (the basis of several agent benchmarks and verifier training sets) measures symptoms rather than causes, then automated agent debugging and self-correction products may deliver less reliability improvement than their evaluation scores imply. That is a hypothesis for future work, not a measured market effect.

Technical Capability: Builders of agent controllers, verifiers, and failure-attribution benchmarks should care because the standard evaluation methodology (judge-labeled first mistakes) is shown to be uncorrelated with the quantity controllers actually need, and a measurement-based alternative is released with code.