Verifier-Induced Support Reshaping in On-Policy Optimization

Reinforcement learning with verifiable rewards (RLVR) on one task reshapes the sampling distribution for a second task primarily at the first response token, making successful trajectories for the other objective too rare to sample and reinforce, even without broadly erasing the underlying ability.

Bidirectional training experiments on Qwen3-8B-Base and Qwen2.5-Math-7B: Math-RLVR trained on 7.5k MATH, IF-RLVR trained on IFTrain, evaluated on AIME24, AIME25, MATH-500-128, IFEval, IFBench, MathIF, and ReasonIF. Cross-task support measured via pass@1 and best@32 over k=32 repeated rollouts per prompt. Sequential training in both orders tested. Distributional shifts localized with Jensen-Shannon divergence by token position under teacher forcing. Opening-route interventions tested causally.

On IFEval with Qwen3-8B-Base, Math-RLVR raises pass@1 by +6.5 pp but lowers best@32 by -9.8 pp. On Qwen2.5-Math-7B the divergence is larger: pass@1 +7.9 pp, best@32 -11.4 pp. IF-RLVR steadily lowers AIME best@k at every budget from k=4 to k=32. In sequential IF-then-Math training, the mixed (learnable) prompt group collapses from 33.6% to 1.6% within 20 steps. KL penalty up to beta=0.12 in Math-then-IF training retains math pass@1 but yields the smallest IF gains; no tested coefficient achieves both. Distributional JS divergence peaks at token position 1 across all conditions.

Each training configuration uses one fixed-seed run, so results reflect within-policy sampling variation rather than variance across training runs. The observed support collapse after IF-first training applies only to 20 observed steps and does not imply irreversible loss of reasoning ability. Opening-route classification is behavioral and does not capture internal reasoning states. Mitigation methods (routing priors, reference-policy constraints, on-policy distillation) are tested in limited configurations. Results are restricted to two model families and two task pairs.

A fair skeptic would note that single-seed runs make it impossible to distinguish systematic effects from training-run variance. The 20-step window for sequential training is short and may not represent long-run dynamics. The finding that support collapses at the first token is consistent with superficial formatting shifts rather than deep capability interference, and longer or differently scheduled training pipelines could bypass the observed bottleneck entirely. The paper also does not test whether the support loss is recoverable with more compute or different data mixtures.

In the 12-24 month horizon, this finding adds a concrete mechanism-level caution for enterprises and labs building multi-stage RLVR pipelines: sequential fine-tuning on verifiable rewards may silently degrade future trainability for adjacent tasks, raising the hidden cost of multi-objective post-training. If confirmed at scale, it complicates the prevalent practice of stacking reasoning and instruction-following fine-tuning stages and may require additional mitigation compute or rearchitected training curricula, modestly increasing per-model development cost. The immediate economic impact is limited, as this is a technical report from a single lab without large-scale validation.

Technical Capability: ML teams building multi-stage RLVR pipelines need to know that optimizing one verifier can silently reduce future trainability for a second objective, and that the bottleneck concentrates at the first generated token, pointing to where mitigations should be targeted.

Market & Valuation: AI labs competing on joint math-and-instruction-following capability face a non-obvious training-order tradeoff that cannot be resolved by simply stacking more RLVR stages, which has implications for product differentiation and training cost estimation.