Learned Reporting Preferences in RLVR Can Conflict with the Current Request

2609.36587 ↗ · 2026-09-29 · cs.LG · h-max 0 · 0 citations

Yupeng Chang, Wenxuan Zhang, Yuan Wu

The finding

Headline claim

RLVR training that rewards one reporting convention can cause a model to lose its pre-existing ability to report answers in a different requested convention, and convention-matched benchmark gains can overstate gains in the answer the model actually commits to.

The test

Crossed design on GSM8K (1,319 test items): Qwen2.5-7B (plus 1.5B, SmolLM2-1.7B, OLMo-2-7B) trained 100 GRPO steps under hash vs boxed conventions, five paired seeds, evaluated under both requested conventions against a shared initial policy, with fixed paraphrase checks, content-matched SFT marker interventions, multiple readers, and independent human calibration in three settings.

Effect size & scope

On GSM8K, boxed-format RLVR reduces requested hash-payload presence by 35.33 to 74.37 percentage points from a 95.45% baseline in four of five Qwen2.5-7B seeds (the fifth improves 2.50 points); changing only the SFT marker reverses convention preference across three seeds; in three human-calibrated settings, convention-sensitive score gains exceed committed-answer correctness gains.

Limitations

Authors concede the effect is non-universal (one seed improved), seed-dependent in magnitude and direction, measured mainly on GSM8K and small models (7B and below), that the accounting decomposition is descriptive not causal, that human calibration leaves ambiguous commitments unresolved, and that coverage conclusions are reader-dependent with nonsignificant comparisons establishing neither loss nor equivalence.

Counterpoint

Small models on one math benchmark with short 100-step training may not generalize to frontier models or production post-training pipelines; the conflict is partly a prompt-sensitivity artifact that real deployers can surface with a single crossed prompt test; five seeds with two fixed post-hoc is a small n and the headline range spans a wide, seed-noisy interval.

Economy implication

Little or nothing yet for the AI economy in the next 12 to 24 months; the direct implication is narrower: enterprise buyers integrating model outputs into parsers or downstream interfaces should convention-test post-trained checkpoints rather than trust benchmark deltas, a modest evaluation-cost line item rather than a market-moving result.

Who this matters for

Technical Capability: RLVR practitioners get a concrete new failure mode and a cheap crossed-evaluation protocol that catches regressions convention-matched benchmarks hide.

Market & Valuation: If benchmark gains partly reflect format-learned scoring artifacts rather than committed-answer improvement, reported model-improvement deltas used in procurement and vendor comparisons need a caveat.

Bears on (advisory · papers never write evidence)

[1.2] Benchmark gains are increasingly contaminated / saturated, overstating progress.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a prominent approach for improving language-model performance on reasoning tasks using automatically checked answers. Yet convention-matched evaluation cannot reveal whether reinforcing one reporting convention reduces adherence to a different request that the initial policy already follows. To test this, we train matched policies under two reporting conventions and evaluate each policy under both current requests, using the same initial policy as a shared reference. We complement this crossed design with controlled interventions and independent human calibration. On GSM8K, boxed-format RLVR reduces the fraction of Qwen2.5-7B responses containing the requested hash-format payload by 35.33--74.37 percentage points relative to a 95.45% initial baseline in four of five training seeds; the fifth improves by 2.50 points. In the four deteriorating runs, almost every response that omits the requested payload instead retains the trained boxed convention, and the same four seeds deteriorate under two fixed paraphrases. Changing only the final-answer marker in supervised targets reverses which reporting convention the model prefers across three seeds, providing controlled evidence that this preference is learnable. Across three settings with independent human calibration, gains under a convention-sensitive scorer exceed the corresponding gains in committed-answer correctness, i.e., the correctness of the answer the model actually commits to. Together, these results separate three distinct post-training outcomes: learned reporting preference, current-request adherence, and committed-answer correctness. They show that convention-matched accuracy alone does not fully characterize post-training behavior and motivate evaluating current-request adherence alongside convention-matched task accuracy.