EvoProbe: Evaluation-Driven Benchmark Evolution for LLM Agents

2610.19433 ↗ · 2026-10-14 · cs.AI

Qianhong Guo, Wei Xie, Xiaofang Cai, Shuoyoucheng Ma, Chengxin Hu, Baosheng Wang

The finding

Headline claim

Benchmark tasks regenerated from model diagnostic feedback are measurably harder for LLM agents than seed-derived tasks, with mean Outcome scores dropping 11.8% and Pass^3 dropping 40.0% (10.42% to 6.25%) across eight frontier models between two rounds.

The test

66 validated cybersecurity CTF tasks (48 Round 1, 18 Round 2) generated from Cybench seed tasks by eight Examiner models, solved by eight models (GPT-5.6-Terra/Luna, GPT-5.5, Claude-Sonnet-5/4.6, DeepSeek-V4 variants, Qwen-3.7-Plus) using a fixed Terminus-2 Bash agent, 3 runs per task, 20 minutes and 30 steps per run, with multi-judge trajectory evaluation and Borda-weighted judge aggregation.

Effect size & scope

Round 1: mean Outcome 4.23-7.74 across models (best GPT-5.6-Terra 7.74), mean Pass@3 52.34%, mean Pass^3 30.47%. Cross-round with solver and examiner configuration fixed, Outcome fell 11.8% and Pass^3 fell from 10.42% to 6.25%. Scope: cybersecurity CTF tasks only, small R2 set (18 tasks from 3 examiner models), one solver harness, results text truncated before full R2 breakdown.

Limitations

Authors concede R2 is much smaller (18 tasks, 3 examiner models), scores depend on judge choice (they mitigate with multi-judge weighting), and some candidate models were excluded for refusing security tasks. Unstated: Pass^3 of 6-10% means very few consistent successes, so the 40% relative drop is on a tiny base; difficulty filtering uses a single designated model (DeepSeek-V4-Flash-Preview) as the difficulty floor; generated tasks were human-reviewed but distribution shift versus real cybersecurity work is unquantified; no statistical significance tests reported; truncated text leaves whether 'harder' means 'better-targeted' or just 'different/noisier' untested.

Counterpoint

A fair skeptic would say the difficulty increase may partly reflect LLM-generated task noise or brittleness rather than better probing of capability boundaries: tasks designed by the same model family that solves them risk circularity, 18 R2 tasks cannot support an 11.8%/40.0% claim with confidence intervals, and the multi-judge LLM evaluation adds subjective variance. This is also lab-adjacent academic work with self-built infrastructure; the key comparison (same tasks, regenerated variants) is not a controlled paired experiment.

Economy implication

Little or nothing yet in direct economic terms. The paper is methods work for evaluation. Its indirect relevance is that if benchmark saturation and contamination are real, reported agent capability gains overstate deployable reliability, which matters for how enterprise buyers should discount agent-benchmark claims when estimating ROI. That translation is untested here.

Who this matters for

Technical Capability: Provides a concrete pipeline for diagnosing where agents actually break down (Reconnaissance 89.12% average vs Flag Capture 41.58%), showing the gap between peak performance (Pass@3 81.25% for the best model) and consistent unattended completion (Pass^3 52.08%).

Market & Valuation: If agent benchmarks are saturating and contaminating, headline scores overstate near-term agent reliability, which should temper revenue projections built on benchmark-driven capability narratives.

Bears on (advisory · papers never write evidence)

[1.2] Benchmark gains are increasingly contaminated / saturated, overstating progress.

[1.3] Agentic reliability (multi-step completion without human correction) is improving materially, not just peak single-shot.

Abstract

Static benchmarks for large language model (LLM) agents face saturation and data contamination risks, motivating continual updates to evaluation tasks. However, aggregate scores and success rates offer limited insight into specific problem-solving bottlenecks and provide limited guidance for subsequent task design. We propose EvoProbe, an evaluation-driven benchmark evolution framework that integrates task generation, diagnostic evaluation, and feedback-driven regeneration into a closed loop. It combines verifiable subtask outcome scoring with evidence-supported multi-judge trajectory evaluation and distills phase-level bottlenecks, success patterns, and failure patterns into diagnostic feedback for generating new executable tasks. Across two rounds of cybersecurity experiments, evaluations of eight models on 66 tasks reveal phase-level performance differences and common failure patterns beyond aggregate scores. In a cross-round comparison with the Examiner model set and solver configuration held fixed, all eight models achieve lower Outcome scores on the updated tasks, with relative decreases of 11.8\% in mean Outcome and 40.0\% in mean Pass\textasciicircum{}3. These results support using fine-grained diagnostics to guide iterative task updates and probe agents' current capabilities and limitations.