Ecdysis: Efficient and Effective Training of Runtime Harnesses for LLM Agents

2609.11677 ↗ · 2026-09-10 · cs.SE, cs.AI

Ruiqing Yue, Yu Cui, Zhuoyu Sun, Sicheng Pan, Xianhong Xue, Tingyu Li, Ting Li, Wenzhuo Zhu, Yi Chen, Yifei Liu, Baohan Huang, Zhe Cui +2

The finding

Headline claim

Training runtime harnesses with batch-level cross-task failure aggregation and multi-role refinement (Ecdysis) evolves agent harnesses faster and more accurately than per-failure iterative evolution, because it separates systematic harness repair from model-specific accommodation.

The test

Experiments across multiple LLMs (including six Qwen3 model-domain combinations in the motivating analysis) and multiple benchmarks, comparing harness-training speed and resulting agent accuracy against existing harness-evolution baselines, plus ablations, training-data curation, and manual analysis of the model-accommodation ratio during evolution.

Effect size & scope

Up to 1.84x speedup in harness training and up to 18.56% improvement in reasoning accuracy of the resulting harness versus existing evolution methods; a fixed human-configured harness reaches 50.28% accuracy on the Qwen3 analysis versus 29.72% with no harness and 43.33% with naive serial agent-driven harness modification; comparable performance to full-data training using only one-quarter of the training data. Scope: benchmark task suites with fixed task models; the text shows peak comparisons across several LLM/benchmark pairs, not uniform gains everywhere (gains are reported as 'up to').

Limitations

The authors concede that cross-task failure recurrence is used as an inductive bias rather than causal proof, that candidate harnesses are accepted only if aggregate training score improves (which can mask per-task regressions), and that harness evolution can overfit to observed tasks. Unstated limitations: results depend on the fixed evaluation framework's scoring; the multi-role diagnosis adds its own LLM API costs and its gains may be prompt- or model-specific; harnesses trained on one model may still be tuned to that model family; only benchmark tasks, no evidence on messy production workloads; full results tables and the latter half of the paper (efficiency, end-to-end timing, cost analyses) are not in the provided text.

Counterpoint

A fair skeptic would note that gains measured on training-like benchmarks risk being benchmark overfitting dressed as generalization; the 18.56% and 1.84x are best-case figures whose typical-case values are unstated; the FDCR multi-role debate may simply be more inference-time compute per training round, so the fairer baseline is matched-compute serial search; 'cross-LLM generalization' is asserted but the provided text gives no transfer numbers; and no external replication exists for an academic-lab self-modifying-agent system.

Economy implication

Little or nothing yet for firm-level economics; the relevant signal is directional: if harness quality is a trainable, automatable layer, some of the value that now accrues to orchestration vendors building hand-tuned agent scaffolds could be compressed, but this paper offers only benchmark evidence, not production ROI evidence.

Who this matters for

Technical Capability: Shows that the harness around a fixed model is a large and trainable lever (harness alone moves accuracy from ~29.7% to ~50.3%), and that smarter failure diagnosis beats naive search in both speed and generalization.

Labor & Knowledge Work: Automating the 'glue engineering' of agent scaffolds suggests the tooling layer around LLMs, currently human-intensive, is itself a target for automation over the next couple of years.

Bears on (advisory · papers never write evidence)

[1.3] Agentic reliability (multi-step completion without human correction) is improving materially, not just peak single-shot.

[1.4] Inference-time compute yields gains that don't saturate quickly on hard reasoning.

Abstract

Self-evolving runtime harnesses can substantially improve the capabilities of large language model (LLM) agents and provide a promising paradigm for optimizing agent execution. Existing harness evolution methods typically rely on iterative search, repeatedly evaluating and revising candidate harnesses based on execution feedback from task instances. While this paradigm enables continuous harness optimization, it incurs substantial time overhead due to repeated agent executions and code modifications, and may overfit to observed tasks and specific failure patterns, resulting in degraded generalization to unseen tasks. We identify the lack of principled failure diagnosis as a key bottleneck in harness evolution: an observed failure can reflect either model-specific deficiencies or systematic harness deficiencies, and directly optimizing against individual failures can lead to unnecessary model-specific accommodation. We therefore propose Ecdysis, an efficient and effective framework that distinguishes model-specific accommodation from harness-level repair and biases adaptation toward systematic harness deficiencies by identifying recurring cross-task failure patterns. Ecdysis adopts a batch-level cross-instance failure aggregation paradigm to jointly analyze failure evidence from multiple task instances and further introduces Failure-Driven Collaborative Refinement to diagnose failure causes and iteratively refine harness modification specifications. By combining cross-instance failure analysis with multi-role diagnosis, Ecdysis enables more effective harness evolution with lower training time. Experiments show that Ecdysis achieves up to a 1.84x speedup in harness training compared with existing harness evolution methods, while improving the reasoning accuracy of the resulting harnesses by 18.56%.