Scrouting: Cost-Aware Routing of Coding Agents by Scouting the Repository First

2608.04804 ↗ · 2026-08-05 · cs.SE, cs.AI · h-max 4 · 0 citations

Ishaan Bhola, Adithyan Krishnan, Mukunda NS

The finding

Headline claim

A small model can scout the repository first, and that verified handoff lets a routed pool of frontier coding agents match the best single model's solve rate on SWE-bench Pro Python at about one fifth the total cost per solve, though the paper's own ablation shows the routing decision itself adds nothing on this benchmark.

The test

Full Python slice of SWE-bench Pro (266 tasks) under the benchmark's official capped budget tier, with four frontier fixers (GPT-5.2, Claude Opus 4.6, Gemini 3 Flash, Kimi K2.5), a 7B searcher trained on 19,905 supervised trajectories, sandbox verification of handoff claims, and a paired calibration study on 100 fresh tasks; plus a zero-cost replay audit of learned routers over public per-task results on three SWE benchmarks.

Effect size & scope

SuperScout solves 159 of 266 tasks versus 158 for the best solo model, at $0.230 total cost per solve versus $1.274, above the random blind-mixing line; searcher GPU cost is under half a cent per task. Solve-set nesting across benchmarks is 0.941, 0.912, and 0.773. But the no-router ablation (always cheapest fixer with handoff) ties the routed system, and the calibration's per-fixer effects are directional only at N=99.

Limitations

Authors concede the routing component does not beat always-choosing-cheapest on the benchmark, the calibration N=99 gives only directional per-fixer effects, most handoff reproduction claims are false and must be stripped, and results are on one benchmark slice in Python. Unstated: costs reflect list API prices and their own accounting, the fixer pool is fixed at four models, and the fresh-task calibration does not demonstrate the headline cost saving transfers outside the benchmark.

Counterpoint

This is a lab's own system evaluated on its authors' chosen benchmark tier with no external replication, and the headline cost claim is carried by the handoff, not the routing novelty the title advertises; prompt and handoff-format sensitivity, the small calibration sample, and possible benchmark contamination of the training trajectories (success-filtered public SWE traces) all leave room for the effect to shrink on real enterprise issue streams.

Economy implication

If the handoff effect replicates, the near-term economics of agentic coding shift toward small specialized models doing cheap localization to cut frontier-token spend per resolved issue, pressuring frontier API pricing; but with the router itself showing no gain on the benchmark, the honest 12-24 month read is that verified sub-task delegation, not smart routing, is the only demonstrated saving.

Who this matters for

Technical Capability: Shows that a 7B model's verified localization artifact can substitute for frontier tokens on repository tasks, and that solve sets across frontier models are nearly nested so accuracy routing has no headroom.

Market & Valuation: If a fifth-cost scout-plus-cheap-fixer stack generalizes, it compresses what enterprises need to pay per resolved issue, which bears directly on frontier-lab pricing power and app-layer margins.

Bears on (advisory · papers never write evidence)

[3.1] Inference cost per unit of capability is falling rapidly.

[5.2] Open-weight models reach "good enough" parity for most economic use cases.

[5.3] API switching costs are low (customers multi-home and swap easily).

Abstract

Frontier language models can resolve repository-level software issues, but each attempt is expensive, and existing routers select a model from the issue text alone. We present SuperScout, which routes after scouting the repository: a 7B searcher, SuperScout-7B, first explores the repository and produces a structured handoff whose reproduction claims are sandbox-verified, with false claims stripped before delivery. The searcher's hidden states, together with the task text, then feed a resume-based router that dispatches the task to one of four frontier fixers. Adding a new fixer requires no retraining. On the full Python slice of SWE-bench Pro (266 tasks) under the benchmark's official capped budget tier, SuperScout matches the best single model's solve rate (159 of 266 for SuperScout, 158 for the best model) at about a fifth of the total cost per solve, and the reported configuration sits above the random traffic-splitting baseline. A no-router ablation, always the cheapest fixer with the handoff, ties the routed system on this benchmark, so the handoff rather than the routing decision carries the result. A paired calibration study points to the mechanism: the handoff appears to redistribute rather than add solving ability, lifting the three cheaper fixers while slightly hurting the strongest, though at $N=99$ the per-fixer effects are directional only; the searcher's hidden states improve cost routing on the calibration labels while the handoff's own text does not. The searcher's compute adds less than half a cent of GPU time per task.