DeLoop: Repeated Actions as Self-Supervision for Long-Horizon Agents

Agents repeatedly re-issue identical tool calls whose observations they already have, and a self-supervised DPO method trained on the agent's own loop recoveries cuts this behavior sharply while reducing steps per task without hurting, and on larger models improving, task success.

CodeGym tool-use benchmark with Qwen3-8B/14B/32B, Llama-3.1-8B, and Gemini models under context compression (summarization at 4,096 and 2,048 tokens) versus full history; loop rates measured via an exact action-observation repeat detector; preference pairs mined from the agent's own rollouts and trained with LoRA DPO (8,000 pairs for Qwen3-8B); ablations include alternative compressors, memory policies, GRPO variants, inference-time blockers, and transfer to other agent environments and benchmarks.

On Qwen3-8B under compression, loop rate falls from 57.3% to 6.1% of tasks with roughly one third as many steps per task and similar success (25.1%); compression itself raises looping from 29.0% (full history) to 57.3%, and 2,048-token compression reaches 64.7%. On Qwen3-32B, loops fall from 38.5% to 12.9% and success rises from 37.7% to 44.8%; Qwen3-14B success rises from 45.1% to 52.3%. DeLoop-Mix lifts Qwen3-8B success to 30.3%. Looping runs fail more often (59.5% average) than loop-free runs (33.4%) and cost about 10.5 extra actions and 11k extra tokens per task. A GRPO per-step loop penalty achieves 15.0% loops versus DeLoop's 6.1%.

Results are concentrated on CodeGym and tool-call agents; a repeated call with a changed observation does not count as a loop, so the definition may miss harmful near-repeats and may flag legitimate re-checks; success gains on small models are modest or absent without cross-domain data; training pairs require the agent to recover from the loop within the rollout; the compressor itself is unchanged and even trained agents still loop sometimes under aggressive compression; the claimed Code/data availability is promised, not yet verified; general-capability preservation is asserted via appendix checks only.

CodeGym is a single synthetic tool environment and the strongest headline number is on an 8B model where task success did not improve, so the method may mostly reduce wasted steps rather than make agents meaningfully more capable; the loop detector's exact-match rule is easy to game and may be brittle across environments with nondeterministic observations; DPO on self-mined pairs can amplify whatever shortcut the agent already has, and gains may not transfer to frontier-scale models where the failure mode is less severe; success deltas on larger models (about 7 points) could be within run-to-run variance given no reported confidence intervals.

Little or nothing yet for aggregate economics; the relevant signal is narrow but real: if long-horizon agents waste a third of their interaction steps on loops, then policy-level fixes that cut step counts materially lower inference cost per completed task, which feeds the falling-cost-per-useful-output story rather than the revenue story. Watch whether this repair holds on production coding and browsing agents, not benchmarks.

Technical Capability: Agent builders get a label-free training signal from a failure mode that compression, blockers, and better memory policies demonstrably fail to fix, with the agent still proposing already-executed calls in 71.3% of tasks even with a duplicate blocker.

Market & Valuation: If agentic products' unit economics depend on steps per completed task, a method that cuts steps to a third with free self-mined supervision is a direct inference-cost lever for anyone selling long-horizon agent work.