training dynamics

On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamicsarxiv.org

The paper systematically evaluates on‑policy vs off‑policy rollouts in LLM distillation, finding that rollout policy matters less than the KL direction and learning rate. Forward KL is robust to rollout choice, while reverse KL prefers student rollouts, and learning rate drives forgetting and sparsity.