TRACE: Training Reasoning Agents for Causal Exploration with Synthesized Rewards

2609.10315 ↗ · 2026-09-09 · cs.AI, cs.LG

Rui Sun, Zhan Shi, Bing He

The finding

Headline claim

Training on synthesized rewards from a simulator (hidden injected causes as oracle labels) lets an open-weight 35B model beat frontier closed models on multi-step diagnostic root-cause attribution, showing reward access matters more than model scale for this task class.

The test

A simulated digital-advertising diagnostic environment with 12 cause types; agents investigate via SQL/Python tool calls on a held-out 235-episode test set, scored by FullAttr@1 (cause plus segment attribution). Qwen3.5-35B-A3B was SFT'd then RL-trained on synthesized rewards.

Effect size & scope

Claude Opus 5 prompted baseline: 0.686 FullAttr@1. Qwen3.5-35B: 0.159 base, 0.637 after SFT, 0.757 after RL; prompted Qwen3.5-122B-A10B only 0.283. Trained policy uses 11.73 tool calls vs 22.05 for prompted base. Holds only within this simulator-generated distribution; generalization to real operational data is untested.

Limitations

Entirely synthetic environment with a fixed 12-cause taxonomy and known signatures; the oracle verifier filters episodes for detectability, so the task distribution is engineered to be solvable; single base model family; evaluation on one 235-episode held-out set; answer-parser sensitivity addressed only in appendix. No real-world campaign data validation. Authors concede nothing beyond this in the visible text.

Counterpoint

The model was trained on the same simulator and cause taxonomy as the test set, so RL gains may partly reflect learning the generator's statistical fingerprints rather than general diagnostic reasoning; frontier baselines are zero-shot prompted with no adaptation; a 12-cause closed taxonomy invites signature matching despite authors' claims; one environment, one seed, no error bars.

Economy implication

Little or nothing yet for the AI economy directly, but it is a template for how firms with domain simulators could cheaply create objective reward signals and specialize mid-size open models to outperform frontier APIs on narrow diagnostic tasks, which would pressure per-token pricing in vertical analytics within 12-24 months if the recipe transfers to real data.

Who this matters for

Technical Capability: Evidence that RL with engineered verifiers extends beyond math/code into ambiguous diagnostic domains, and that reward access can dominate model scale.

Market & Valuation: If vertical-domain specialization beats frontier models, app-layer firms with proprietary simulators may retain value, informing where rents settle.

Labor & Knowledge Work: Automating root-cause diagnosis, a skilled analytics task, hints at which data-analyst work becomes trainable and automatable, though with synthetic evidence only.

Bears on (advisory · papers never write evidence)

[1.3] Agentic reliability (multi-step completion without human correction) is improving materially, not just peak single-shot.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has advanced language-model reasoning in domains such as mathematics and code, where objective answers are inexpensive to check. Diagnostic reasoning over complex data lacks this advantage: establishing the true cause of an anomaly often requires costly expert investigation and may remain ambiguous after the fact. We ask whether this asymmetry of verification can instead be engineered. We sample an intervention, inject it into a controlled simulator, and generate the observations it would produce. The hidden intervention provides an oracle label and objective reward, while the agent must still investigate noisy, confounded, and distributed evidence. We instantiate this approach in TRACE, a digital-advertising diagnostic environment with 12 root causes and fine-grained segment attribution. Agents investigate each episode using Python and SQL and must identify both the root cause and, when applicable, the affected segment assignment. On a held-out 235-episode test set, the strongest prompted baseline, Claude Opus 5, reaches 0.686 FullAttr@1. Supervised fine-tuning raises Qwen3.5-35B-A3B from 0.159 to 0.637, and subsequent RL with synthesized rewards reaches 0.757, outperforming all evaluated prompted baselines, including frontier closed-source models and a prompted Qwen3.5-122B-A10B model. The resulting policy also uses substantially fewer tool calls than the prompted 35B base. These results provide evidence that access to a scalable, objective training signal can be a more important constraint than model scale alone. More broadly, simulation-based verification can make otherwise ambiguous diagnostic reasoning tasks amenable to scalable reinforcement learning.