Hugging Face Daily PapersXiuYu Zhang, Wei Chow, Junfeng Fang1 min readpaperadvanced
What Does Privileged Information Add to On-Policy Self-Distillation?
Summary
The paper isolates the contribution of privileged teacher information in on‑policy self‑distillation (OPSD) using a 5k‑problem math suite. Reference‑free distillation explains most gains, with only modest extra benefit from full solution traces.
- Reference‑free OPSD accounts for the bulk of Qwen3‑1.7B's improvement; privileged references add only modest gains.
- Polished solution references give small benefit, while full reasoning traces add ~2 % accuracy for SmolLM3‑3B at step 50.
- Switching from short direct‑response rollouts to long thinking‑enabled rollouts can turn gains into losses, even with the same data.
- Altering token‑level teacher supervision (teacher profiles) has little impact on the student model's behavior.
LLM researchers and engineers designing self‑distillation pipelines need to know that privileged references yield limited extra performance, guiding data and teacher design choices.
6/10
