Hugging Face Daily PapersLanglin Huang, Hao Liu, Mononito GoswamiSep 29, 20261 min readpaperadvanced

On the Off-Policy Teacher in On-Policy Distillation

Summary

On-policy distillation (OPD) faces a challenge where the teacher's performance degrades when supervising student-generated, off-policy trajectories. The SCOUT framework addresses this by co-training the teacher to adapt to student-generated prefixes using reinforcement learning, consistently improving OPD effectiveness.

OPD teachers degrade when supervising student-generated, off-policy trajectories.
Teacher continuation performance worsens as student-generated prefixes lengthen.
SCOUT is a co-training framework that adapts the teacher to student prefixes.
SCOUT uses RL with verifiable rewards to optimize the teacher's conditional ability.

Engineers working on improving large language model efficiency and performance through distillation will find this relevant for addressing a core limitation in current OPD methods.

8/10

llm machine learning reinforcement learning distillation model training

Related reading

Calibrating Teacher--Student Discrepancy for On-Policy Distillation arxiv.org
This paper introduces Calibrated On-Policy Distillation (Cal-OPD), a method to improve knowledge distillation by filtering out the teacher model's inherent deviations from the learning signal. Cal-OPD estimates and removes the teacher's self-deviation, leading to better student performance on mathematical reasoning tasks.
Hugging Face Daily Papersarxiv.orgQiangqiang He, Jin Li, MingCai Chen18d1 minpaper
On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamics arxiv.org
The paper systematically evaluates on‑policy vs off‑policy rollouts in LLM distillation, finding that rollout policy matters less than the KL direction and learning rate. Forward KL is robust to rollout choice, while reverse KL prefers student rollouts, and learning rate drives forgetting and sparsity.
Hugging Face Daily Papersarxiv.orgJulianna Piskorz, Antonin Berthon, Mihaela van der Schaar8d1 minpaper
RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning arxiv.org
RetireOPD introduces a self‑retiring on‑policy distillation framework for multi‑turn RL agents. A skill‑conditioned teacher is first trained with environment rewards, then a skill‑free student learns jointly via RL and token‑level distillation. The student automatically drops the teacher once its performance gap stops shrinking and it reaches a target success‑rate fraction, after which training c…
Hugging Face Daily Papersarxiv.orgYan Yu, Zhengxi Lu, Yizhou Liu19d1 minpaper
Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation arxiv.org
The paper shows that standard multi-teacher on-policy distillation (MOPD) gives unbalanced updates, hurting specialist knowledge like math. By measuring each domain’s feedback spread and normalizing it (DN-MOPD), they rebalance updates, recover most math performance, and improve average scores on six benchmarks.
Hugging Face Daily Papersarxiv.orgXin Li, Hao Jiang, Xin Gao8d1 minpaper
When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation arxiv.org
On-policy distillation (OPD) can lead to excessively long student responses, a phenomenon called length inflation. This paper identifies "termination-token mismatch" between base students and post-trained teachers as a key source, where models place stopping probability on different EOS tokens. Treating functionally equivalent EOS tokens as a shared semantic stopping action substantially mitigate…
Hugging Face Daily Papersarxiv.orgYuxiao Yang, Tianrun Yu, Shangzhe Li19d1 minpaper
LastOPD: Taming Collapse in Latent On-Policy Distillation arxiv.org
The authors find that latent supervision in on‑policy distillation can cause a dramatic performance collapse even as alignment improves. Their LastOPD method limits latent alignment to the final‑layer state and crossfades to token‑level supervision, delivering 4–5 point MATH‑500 gains and halving the steps to match token‑only OPD.
Hugging Face Daily Papersarxiv.orgJie Yang, Zhengyu Fang, Zelin Xu13d2 minpaper

On the Off-Policy Teacher in On-Policy Distillation

Summary

Related reading

Calibrating Teacher--Student Discrepancy for On-Policy Distillationarxiv.org

On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamicsarxiv.org

RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learningarxiv.org

Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillationarxiv.org

When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillationarxiv.org

LastOPD: Taming Collapse in Latent On-Policy Distillationarxiv.org

Calibrating Teacher--Student Discrepancy for On-Policy Distillation arxiv.org

On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamics arxiv.org

RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning arxiv.org

Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation arxiv.org

When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation arxiv.org

LastOPD: Taming Collapse in Latent On-Policy Distillation arxiv.org