DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Training

2609.04094 ↗ · 2026-09-03 · cs.AI, cs.LG, cs.SE

Shubham Gandhi, Saurabh Goyal, Kiran Kate, Yara Rizk

The finding

Headline claim

Reinforcement learning for long-horizon tool-using agents can work without any ground-truth verifier by scoring dynamically generated rubrics once per trajectory with an LLM judge and redistributing that score across steps, matching or beating verifier-based RL on AppWorld.

The test

GRPO training with LoRA on the 90-task AppWorld training split using Qwen3.6-27B and Qwen2.5-32B-Instruct on 8 H100s, judged by GPT-5.4; evaluation on held-out AppWorld TN (168 tasks) and TC (417 tasks) and zero-shot transfer to tau-bench Banking, compared against static-rubric, no-step-credit, and outcome-reward (verifier-based) ablations.

Effect size & scope

On AppWorld TN, DRACO reaches 85.3 TGC (p1), a 15.9-point gain over the base model and 5.3 points over GRPO with sparse ground-truth reward despite using no verifier; on out-of-domain tau-bench it gains 5.3 points over the base model even with a self-judge. Scope: one open-weight model family, one training domain (90 tasks), LoRA fine-tuning only, and a frontier model (GPT-5.4) as judge during training.

Limitations

The authors concede a dependence on judge quality (they test a self-judge variant that underperforms), evaluation cost of rubric generation and scoring per rollout, and that rubric criteria are mostly task-specific sub-goals with only static criteria recurring. Unstated: the judge is a far more capable frontier model than the trained policy, so the method may implicitly distill frontier judgment rather than learn outcome competence; risk of rubric gaming by the policy as it adapts to judge preferences is unquantified; single-benchmark training means generality beyond AppWorld-style tool use is untested; train/eval contamination via the same judge model used as tau-bench simulator and judge.

Counterpoint

A skeptic would note the headline comparison (outcome-blind beats verifier-based RL by 5.3 points) may reflect the frontier judge being a stronger teacher than sparse binary unit-test rewards, not a victory for outcome-blindness; n is effectively one model family and one domain; AppWorld is a well-known benchmark where verifier-reward baselines are cheap and well tuned by others; judge prompts (many, in Appendix F) are a large prompt-sensitivity surface; and rubric-scored rewards are exactly the kind of learned-proxy signal RLVR was designed to avoid gaming.

Economy implication

Little or nothing yet for the economics of AI in the next 12 to 24 months: this is a training-recipe result on academic benchmarks. Its economic relevance is conditional and indirect, namely that if agent RL can be supervised by LLM judges instead of hand-built verifiers, the cost of producing reliable domain agents (customer support, research assistants) falls, but that translation is unproven here.

Who this matters for

Technical Capability: Researchers building RL post-training for agents should care because it removes the verifier requirement that currently blocks RL in most open-ended agent domains and shows step-level credit redistribution is feasible without a learned module.

Labor & Knowledge Work: Teams automating customer-support or tool-heavy knowledge work should care only conditionally: the target domains (support agents, open-ended research) are exactly the occupations exposed to agentic automation, but no deployment evidence appears here.

Bears on (advisory · papers never write evidence)

[1.3] Agentic reliability (multi-step completion without human correction) is improving materially, not just peak single-shot.

[B1] Agentic reliability gains will translate into enterprise ROI.

Abstract

Reinforcement Learning from Verifiable Rewards works well when a task has a programmatic checker, but most long-horizon agent domains have none. We work in the outcome-blind setting, where ground-truth success signals are not available. Multi-criteria rubrics are a popular way to supply such a reward; they are scored once per trajectory, but a single scalar is a poor signal across tens of steps. We propose DRACO: Distributing Rubric-based Advantage for Credit Optimization. It generates rubrics dynamically during training to track the policy's evolving capability, scores those rubrics once per completed trajectory, and redistributes that judgment over the steps responsible for annotated rubrics to produce differentiated per-step advantages in GRPO. The redistribution is closed-form and does not introduce any trained attribution module. On AppWorld, DRACO gains 15.9 points over the base model and 5.3 points over GRPO trained with a sparse ground-truth reward, despite not using any verifiers itself. On out-of-domain Tau-Bench, it gains 5.3 points over the base model even without a frontier judge, beating both ground-truth-reward training and other rubric-based training settings. The code for DRACO is available at https://github.com/IBM/draco.