T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks

2609.11042 ↗ · 2026-09-10 · cs.LG, cs.AI

Junyao Yang, Yucheng Shi, Zhongzhi Li, Ruhan Wang, Zongxia Li, Haitao Mi, Leowei Liang

37 pages, 18 figures

The finding

Headline claim

A 122B MoE model post-trained with reinforcement learning on executable terminal tasks, using TITO and rollout routing replay to stabilize MoE training and a dense per-assertion verifier reward, raises Terminal-Bench 2.1 performance from 43.8% (base) or 49.4% (SFT) to 64.0%, placing it near or above much larger frontier models, and reaches 27.9% on Long-Horizon Terminal-Bench.

The test

Post-training of Qwen3.5-122B-A10B with PPO across three epochs on ~15,000 synthesized terminal tasks executed in real cloud sandboxes (up to 300+ tool-call turns per task), evaluated on three held-out execution-scored benchmarks: Terminal-Bench 2.1 (89 tasks), Long-Horizon Terminal-Bench, and Terminal-Bench Hard, against GPT-5.4, Claude Opus 4.6/4.7, DeepSeek-V4-Flash/Pro and Gemini-3.1-Pro under an identical agent harness.

Effect size & scope

On Terminal-Bench 2.1: base 43.8% to 64.0% resolved, a 28.5% relative gain from RL alone over the 49.4% SFT checkpoint, beating GPT-5.4 (54.8%), DeepSeek-V4-Flash (56.9%) and Claude Opus 4.6 (63.8%), just short of Claude Opus 4.7 (66.1%). On Long-Horizon Terminal-Bench: 27.9%, matching Gemini-3.1-Pro. Terminal-Bench Hard: 38.0%. Domain gains are uneven (100.0% debugging, 88.9% system administration) and concentrated on Medium-difficulty tasks; training-inference log-probability gap cut from 0.021 to 0.013 with zero token drift in the loss region.

Limitations

The authors concede: conditioning is exact only up to re-tokenized history; no single-axis reward ablation exists so the contribution of dense reward versus TITO/R3 is unattributed; verifier integrity is filtered by an LLM audit rather than enforced; the long tail is paid for twice (straggler rollouts plus repeated compute); data-distribution skew (top five categories are 67.9% of the pool, debugging only 1.3%) predicts where residual failures land; staleness/asynchrony and reporting gaps remain; scope is limited to terminal tasks. Unstated: Terminal-Bench 2.1 has only 89 tasks, so headline differences of a few points sit on very small effective samples; the binary-reward campaign never beat its supervised baseline, so the pipeline's headline depends on the dense-reward variant on curated data.

Counterpoint

A fair skeptic would note the 89-task evaluation set makes GPT-5.4 versus T1 differences statistically fragile; synthesized training tasks and the evaluation both come from the RST synthesis lineage, so the claimed out-of-distribution guarantee rests on the authors' own seed-disjointness and LLM audit rather than independent verification; the harness, verifier filtering, and reward design are all self-controlled and could still leak task-specific advantage; comparison numbers for frontier competitors are produced by the authors under one harness with no independent replication; and dense partial-credit rewards invite subtle reward farming that the authors mitigate by monitoring, not proving, absence.

Economy implication

Little or nothing yet for the AI economy in the next 12 to 24 months as a direct read, but one signal matters: a mid-sized open-lineage MoE model (122B total, ~10B active) trained by RL on executed verifiable outcomes reaches near-frontier terminal-agent performance, which, if replicated, supports falling capability-per-dollar at the agent layer and intensifying model commoditization from below.

Who this matters for

Technical Capability: Engineers and researchers building agents should care that the paper shows executable, per-assertion verification rewards plus training-inference consistency fixes are what unlock long-horizon RL, while binary outcome rewards failed to beat the SFT baseline.

Market & Valuation: Investors should note the claimed compression of frontier-versus-mid-size gaps on agentic terminal benchmarks, which bears on how long frontier-lab pricing power lasts.

Labor & Knowledge Work: If terminal autonomy genuinely improves, the software-engineering task surface (debugging, sysadmin, packaging) moves closer to automation, though one benchmark paper is far from labor-market evidence.

Bears on (advisory · papers never write evidence)

[1.3] Agentic reliability (multi-step completion without human correction) is improving materially, not just peak single-shot.

[1.4] Inference-time compute yields gains that don't saturate quickly on hard reasoning.

[5.1] Frontier models cluster tightly (small gaps, fast catch-up).

Abstract

Agent usage is shifting toward long-horizon tasks such as coding and scientific discovery, among which terminal tasks are especially important. We introduce T1, a Mixture-of-Experts model of 122B total trained with reinforcement learning, operating a real shell in a cloud sandbox for up to 300+ tool-call turns per task, rewarded by executing each task's own verifier. We provide a comprehensive recipe: First, an aggressively warm-started to stabilize actor-critic training, with a dense process reward scoring trajectories by the absolute number of passing verifiers. Second, stable optimization through TITO construction, training on the exact sampled token identifiers with drift repair at turn boundaries, and rollout routing replay, recording the sampler's per-token expert choices at every MoE layer and replaying them during training. Third, fully out-of-distribution training corpus: isolated seeds and synthesized tasks disjoint from Terminal-Bench 2.1 ensures gains reflect genuine capability transfer over benchmark overfitting. Together, TITO and R3 cut the training-to-inference log-probability difference from 0.021 to 0.013, with exactly aligned zero token drift in the loss region. On Terminal-Bench 2.1, our post-train pipeline raises initial base model from 43.8% to T1 with 64.0% resolved. On Long-Horizon Terminal Bench, T1 reaches 27.9% and surpasses GPT-5.4 and GLM-5.1.