Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation

2608.04794 ↗ · 2026-08-05 · cs.AI, cs.LG · h-max 3 · 0 citations

Sarthak Harne, Chinmay Karkar, Yash Pandya, Ahmed Awadallah, Akshay Nambi

The finding

Headline claim

Self-distillation with privileged information, used as a lone training objective without any reward term, fails to improve reasoning accuracy on difficult tasks and typically degrades it, because the per-token loss optimizes surface imitation of one reference trajectory rather than correctness.

The test

Qwen3-8B (think and instruct modes) and Qwen3-32B trained with the SDPO and OPSD self-distillation objectives, no reward term, across four domains: general QA (MMLU-Pro), mathematics (DAPO-Math-17k), coding (CodeForces), and multi-turn agentic tool use (BFCL), each with 2,000 training examples, evaluated on in-domain and held-out transfer benchmarks. Training was reproduced in the easy original regime (SciKnowEval) to confirm implementation fidelity, then applied identically to the harder domains.

Effect size & scope

Lone SD degrades average accuracy across domains by up to 3.51 points in the agentic domain and up to 7.0 points on individual benchmarks. PI Bias Score reaches 0.52 for the in-context reference solution versus at or below 0.02 for a different correct solution to the same problem. The loss is nearly identical for correct and incorrect rollouts (per-token loss approximately 3.5e-4, KL approximately 0.03 in both cases). Off-path exploratory tokens carry KL of approximately 0.31 versus approximately 0.08 on-path tokens, a factor of four. Stopwords, punctuation, and uncertainty markers absorb 55.4 percent of the per-token loss. The effect holds across reasoning modes, model sizes (8B and 32B), PI forms (whole solution, hint, skill), and both SDPO and OPSD recipes. The one partial exception is the agentic instruct mode with a small training set and short responses, which shows a modest gain.

Limitations

Authors acknowledge that the easy-setting success of SD is reproducible and do not claim SD is universally broken, only that it fails as a lone objective on difficult tasks. They do not test SD combined with a reward term as a primary signal. The study uses only the Qwen3 model family, so generalization to other architectures is untested. Training sets are capped at 2,000 examples, which may not reflect large-scale industrial use. The PI Bias Score is a diagnostic instrument derived from forward passes only and has not been validated as a prescriptive fix.

Counterpoint

A skeptic would note that the failure is specific to SD as a lone objective, a setting that neither SDPO nor OPSD actually recommends for hard tasks at scale. Both papers use SD alongside or in lieu of GRPO on comparatively easy benchmarks, so the paper is partly testing a strawman. The modest positive results in agentic instruct mode and scattered positive deltas in coding suggest the signal is not uniformly harmful. The choice of JSD with aggressive clipping and a very slow EMA teacher introduces additional hyperparameter sensitivity that could mask or amplify the effect.

Economy implication

SD was positioned as a compute-efficient alternative to RLVR for post-training, potentially reducing the cost of producing capable reasoning models. If SD cannot be used as a standalone objective on hard tasks, practitioners must retain verifier infrastructure and on-policy reward computation, keeping the cost baseline for reasoning-capable post-training higher than SD proponents suggested. In the next 12 to 24 months this raises the effective floor cost for fine-tuning frontier-capable reasoning models, modestly benefiting providers with verifier infrastructure and complicating claims about cheap post-training via distillation alone.

Who this matters for

Technical Capability: Researchers and engineers building post-training pipelines for reasoning models need to know that dense per-token imitation of a privileged reference solution does not substitute for verifiable reward signals on hard tasks, and that the PI Bias Score offers a diagnostic for detecting this failure mode.

Market & Valuation: Vendors and startups that marketed self-distillation as a low-cost path to RLVR-level reasoning gains face a credibility challenge: the efficiency argument breaks down precisely where reasoning capability matters most commercially.

Bears on (advisory · papers never write evidence)

[1.4] Inference-time compute yields gains that don't saturate quickly on hard reasoning.

[3.1] Inference cost per unit of capability is falling rapidly.

[B3] Training-efficiency gains will reduce aggregate compute demand (threatening "more compute = more value").

Abstract

Self-distillation (SD) has emerged as a compute-efficient alternative to reinforcement learning with verifiable rewards: a self-teacher, conditioned on privileged information (PI) about the answer such as a reference solution, supplies dense per-token supervision to a student that never sees it. Reported gains, however, come almost exclusively from narrow, low-difficulty settings, leaving open a basic question: as a lone objective, with no reward term, does SD teach anything? We reproduce SDPO's reported gains in its easy setting, then apply the identical setup to difficult tasks and find that it does not. Across question answering, mathematics, coding, and multi-turn agentic tool use, across reasoning modes, model sizes, and forms of PI, and under both the SDPO and OPSD recipes, the per-token loss falls steadily while validation accuracy does not improve and typically degrades. We explain this failure through a single causal chain from the loss to the model it produces. The chain begins with PI bias: having seen one particular reference solution, the teacher's per-token target is pulled toward that trajectory rather than toward correctness in general, an effect we quantify with a PI Bias Score. Trained to match this target everywhere, the student's objective becomes nearly blind to whether a rollout is correct, and the loss it assigns falls mostly on low-information tokens like stopwords, punctuation, uncertainty markers, rather than those that determine the answer; within correct rollouts the exploratory tokens incur the highest divergence, so it penalizes the hesitation that reasoning requires. The result is a flatter, less decisive student that is no better at reasoning: as a lone objective, SD optimizes a signal decoupled from task success.