Benchmark performance for AI agents routinely overstates reliability in real, unattended task completion, and the mechanisms driving failures in long-horizon tasks remain poorly understood. The evidence base here is currently thin: only one paper directly addresses this thread. The core question, how far eval performance translates to deployment reliability, remains open.
[2607.07702] attacks the eval-to-real gap from a diagnostic angle rather than a benchmark-construction angle. The paper introduces STRACE, a framework that parses agent execution logs into causal dependency graphs and uses backward slicing to pinpoint which specific modules are responsible for task failures. By repairing those root-cause modules rather than applying generic prompt optimization or context filtering, STRACE yields large absolute gains on hard benchmarks: +12.9 percentage points on WebArena ALL (10.8% to 23.7%), +16.0pp on VeruSAGE-Bench ALL (42.5% to 58.5%), and a jump from 37.0% to 68.5% exact match on HotpotQA. The next-best baselines, TextGrad and Summary-based Selection, achieve only +6.5pp and +2.8pp respectively on their respective benchmarks. Ablations confirm that each of the four phases in the STRACE pipeline contributes independently to the gains.
The implicit message of [2607.07702] is important for this thread: agents fail not randomly but through identifiable, structured causal chains in their execution traces. This suggests the eval-real gap is at least partly a diagnostic gap, agents fail in systematic ways that standard evals do not surface, and that targeted structural analysis can recover substantial reliability without changing the underlying model.
With only one paper, there is no conflict to report, but the evidence is narrow in several ways. STRACE is evaluated on benchmarks (WebArena, VeruSAGE-Bench, HotpotQA), not on genuinely unattended real-world deployments, so the question of whether causal-graph-based repair holds up outside structured eval environments is unanswered. It is also unclear how much human effort the STRACE diagnostic pipeline requires in practice, which matters for the "unattended" criterion. The paper does not compare against human-level performance ceilings, so the residual gap after STRACE is applied remains large (e.g., 23.7% on WebArena ALL is still a low absolute number).
STRACE, a framework that models agent execution logs as causal dependency graphs and uses backward slicing to isolate root-cause modules, improves long-horizon agent success rates significantly more than standard context-filtering or prompt-optimization baselines.
On WebArena ALL, STRACE achieves +12.9 absolute percentage points over the base agent (10.8% to 23.7%), versus the next-best baseline TextGrad at +6.5pp. On VeruSAGE-Bench ALL, STRACE achieves +16.0pp (42.5% to 58.5%), a 1.4x success-rate improvement, while the next-best baseline (Summary-based Selection) achieves only +2.8pp. On HotpotQA EM, STRACE reaches 68.5% versus base agent 37.0%. Gains are consistent across all three benchmarks and all sub-domains tested. Ablation confirms each of the four phases contributes to overall performance.
AI agents draw systematically different conclusions from identical numerical data when the substantive framing changes, with conclusions biased toward each agent's prior beliefs and sometimes exhibiting motivated reasoning through selective analysis.
Across all twelve agent-domain comparisons, the rate of affirmative conclusions was highest under the prior-aligned frame and lowest under the prior-opposed frame, with the neutral frame in between. This held for both binary conclusions and point estimates (odds ratios, margins, probabilities). In the medical task, the ground-truth odds ratio was approximately 1.8 to 1.9 under correct specification, but post-exposure covariate inclusion could attenuate this to null or protective, and framing influenced which path agents took. Motivated reasoning evidence was clearest in the medical task and weaker in election and geopolitical tasks. Scope conditions showed that explicitly narrowing agent discretion reduced but did not eliminate framing effects.
Reliable use of progressively disclosed skills by LLM agents remains out of reach: the best model-harness configuration scores only 0.613 on the Skill-Use benchmark, with skill triggering and procedural compliance failing as separate bottlenecks, and skill use behaving as a harness-conditioned capability rather than a fixed model property.
Best SU is 0.613 (GPT-5.5 under Claude Code); Trigger is near-ceiling for closed models (0.94-0.97 under CC) but drops to roughly a third for several open-weight models (DeepSeek-V4-Pro 0.324, Kimi-K2.6 0.337), while even on triggered traces the best Compliance is only 0.638. Boundary consistently exceeds Compliance across all configurations. Rankings shift with harness: GPT-5.5 falls from SU 0.613 under CC to 0.503 under Codex, and several mid-tier models rise. Effects hold across nine domains but the sample is small: one task per skill, 177 total.
Replacing the orchestration harness alone, while holding models and tasks constant, cuts enterprise agentic AI token spend by 38% and cost per task by 41%, with task-completion quality holding at parity or improving.
Blended across six models: cost per task fell 41% ($0.21 to $0.12), median wall-clock fell 44% (48s to 27s), tokens per task fell 38% (14.2k to 8.8k). Quality per dollar rose 82%; task-completions per million tokens rose from 54.9 to 92.0 (+68%). Per-model cost reduction ranged from 33% (Gemini 3.1) to 61% (Gemini Flash 3.5), with no model showing a cost increase. Headline quality was 0.78 to 0.81 (not statistically significant at n=22). All 7 quality regressions were on the three smaller/weaker models, concentrated in orchestration-heavy tasks. Harness leverage (quality gain correlated with baseline model strength) was r=0.99 across n=6 models.