EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction

2609.02783 ↗ · 2026-09-02 · cs.CL

Yuling Shi, Zhensu Sun, Junsen Dong, Chengcheng Wan, David Lo, Xiaodong Gu

Code and data available at https://github.com/inphotoo/earlyeval

The finding

Headline claim

A lightweight pair of LightGBM classifiers can halt agent benchmark runs early with high confidence, cutting 13-26% of execution steps and up to 44% of input tokens while shifting measured resolve rates by only one to two percentage points and preserving leaderboard rankings.

The test

Over 21,000 outcome-labeled trajectories from 16 agents on SWE-bench Verified, 37 on TerminalBench, and 22 on Toolathlon; leave-one-agent-out evaluation where predictors train on historical trajectories of other agents and are tested on a held-out agent, with rankings compared against full-run leaderboards.

Effect size & scope

On SWE-bench Verified, roughly 35% of runs halted at 95% prediction accuracy, 26% of steps eliminated, 33% input and 29% output tokens saved, average resolve-rate shift of 1.1 percentage points, Spearman rho 0.991 across 16 agents. On TerminalBench and Toolathlon, 13-25% of steps saved at 89-97% accuracy, ranking rho at or above 0.959. Scope: applies to evaluation harnesses, not production agent runs, and requires a pool of historical labeled trajectories per benchmark.

Limitations

The authors note the method perturbs individual resolve rates by one to two percentage points, which matters for close comparisons; prediction accuracy of 89% on some benchmarks means some outcomes are mislabeled; benchmarks without gold solutions lose the reference-solution feature family. Unstated: the savings are modest relative to headline cost (26% steps saved on the cheapest benchmark), predictors may be tuned to the scaffolding conventions of OpenHands-style trajectories, and cutting runs short could mask late reversals (agent recovers after apparent failure), biasing scores downward in ways correlated with agent style.

Counterpoint

The per-agent resolve-rate perturbation of one to two points is the same order as the gaps separating adjacent leaderboard entries, so early-stopped evals could misrank models on genuinely close comparisons despite high aggregate rho; accuracy is reported against thresholds the authors choose, allowing a tunable tradeoff to be presented favorably; the approach assumes future agent trajectories resemble the historical pool, which may break as scaffolds and models change; and this is an eval-harness efficiency trick, not evidence about agent capability itself.

Economy implication

Modest near-term: it lowers the marginal cost of the agent development loop, which is real but small next to training and inference costs, and mostly benefits teams that run many benchmark iterations. It says little directly about enterprise ROI or labor effects.

Who this matters for

Technical Capability: Practitioners iterating on agents get a plug-and-play way to cut evaluation bills by a quarter or so without visibly changing leaderboard order, which makes frequent eval cheaper.

Market & Valuation: Cheaper, faster evaluation slightly lowers a barrier for smaller labs and startups competing on agent scaffolds, modestly eroding an iteration-speed advantage of well-funded teams.

Abstract

Evaluating LLM agents is essential for guiding their development, yet it has grown prohibitively expensive: a single pass of a frontier model over an agentic benchmark can cost hundreds to thousands of dollars, a price paid repeatedly across iterative development cycles. Prior efforts, centered on benchmark distillation, reduce the number of evaluation tasks but leave the cost of executing each retained task untouched. In this work, we introduce early outcome prediction, a complementary axis of efficiency that instead cuts cost within each task. Our key insight is that an agent's final outcome is often evident from its intermediate behavior well before execution completes. We instantiate this idea in EarlyEval, a lightweight framework that trains a pair of LightGBM success and failure classifiers over behavioral, textual, and reference-solution features, and halts an agent run the moment either classifier crosses a calibrated confidence threshold, adding negligible per-step overhead. Across three benchmarks, SWE-bench Verified, TerminalBench, and Toolathlon, EarlyEval can eliminate 13%-26% of agent steps and up to 44.1% input tokens and 29.4% output tokens at 89%-97% prediction accuracy, while perturbing per-agent resolve rates by only one to two percentage points on average.