proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersHuanxin Sheng, Zhiling Ye, Haonan Wang1 min readpaperadvanced

1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation

Read on arxiv.org

Summary

The paper studies gradient‑estimation noise in sparse on‑policy distillation (OPD) and introduces the Information‑Efficiency Ratio (IER), a signal‑to‑noise based metric for selecting which tokens to supervise. IER is derived from an information‑geometry analysis with an optimal scalar baseline, and a candidate‑set approximation lets it be combined with existing usefulness scores while keeping the…

  • Gradient estimates from a single sampled next token can be noisy; IER quantifies this noise relative to signal.
  • IER is computed analytically for a given prefix and can be approximated efficiently via a candidate set.
  • Combining IER with prior usefulness scores improves token‑selection for sparse OPD.
  • Empirical results: on math and medical reasoning tasks, 0.1%–1% token budgets with IER‑guided selection achieve performance comparable to or better than dense OPD.

Sparse supervision is crucial for scaling teacher‑student training of large language models, but noisy gradient estimates can degrade learning. By providing a principled, low‑overhead metric to assess the reliability of each token’s gradient, IER enables more efficient allocation of teacher signals…

8/10
llmmachine learningdistillationon policytoken selectiongradient estimation

Related reading

  1. When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation

    On-policy distillation (OPD) can lead to excessively long student responses, a phenomenon called length inflation. This paper identifies "termination-token mismatch" between base students and post-trained teachers as a key source, where models place stopping probability on different EOS tokens. Treating functionally equivalent EOS tokens as a shared semantic stopping action substantially mitigate…

    Hugging Face Daily Papersarxiv.org1 minpaper

  2. Trajectory as the Teacher: Few-Step Discrete Flow Matching via Energy-Navigated Distillation

    Apple researchers propose Trajectory‑Shaped Discrete Flow Matching (TS‑DFM), a training‑time distillation method that replaces blind stochastic jumps in discrete flow‑matching with an energy‑based compass to select higher‑quality intermediate tokens. On a 170 M‑parameter language model, the 8‑step student outperforms the 1 024‑step teacher by 32 % perplexity while being 128× faster, beating basel…

    Apple Machine Learning Researchapple.com1 minpaper

  4. onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction

    onPanda is an interactive annotation tool that lets humans correct LLM outputs token‑by‑token, then resumes generation from the corrected prefix. In a controlled study it cut median annotation time by 52% and the authors release a token‑level correction dataset (Panda‑CVL) for on‑policy fine‑tuning.

    Hugging Face Daily Papersarxiv.org1 minpaper

  6. RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning

    RetireOPD introduces a self‑retiring on‑policy distillation framework for multi‑turn RL agents. A skill‑conditioned teacher is first trained with environment rewards, then a skill‑free student learns jointly via RL and token‑level distillation. The student automatically drops the teacher once its performance gap stops shrinking and it reaches a target success‑rate fraction, after which training c…

    Hugging Face Daily Papersarxiv.org1 minpaper