proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersYuxiao Yang, Tianrun Yu, Shangzhe Li1 min readpaperadvanced

When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation

Read on arxiv.org

Summary

On-policy distillation (OPD) can lead to excessively long student responses, a phenomenon called length inflation. This paper identifies "termination-token mismatch" between base students and post-trained teachers as a key source, where models place stopping probability on different EOS tokens. Treating functionally equivalent EOS tokens as a shared semantic stopping action substantially mitigate…

  • On-policy distillation (OPD) can cause student LLMs to generate excessively long responses, exhausting generation budgets.
  • A primary cause is "termination-token mismatch" where student and teacher models use different EOS tokens for stopping.
  • This mismatch occurs even when models like Qwen3, Llama, and Gemma declare identical stopping token sets.
  • Aligning decoding stopping sets is insufficient; treating functionally equivalent EOS tokens as a shared semantic action is crucial.

Engineers working on LLM distillation or fine-tuning should understand this mechanism to prevent common issues like response length inflation and improve model efficiency.

7/10
llmmachine learningnlpdistillationtransformersfine tuning

Related reading

  2. Trajectory as the Teacher: Few-Step Discrete Flow Matching via Energy-Navigated Distillation

    Apple researchers propose Trajectory‑Shaped Discrete Flow Matching (TS‑DFM), a training‑time distillation method that replaces blind stochastic jumps in discrete flow‑matching with an energy‑based compass to select higher‑quality intermediate tokens. On a 170 M‑parameter language model, the 8‑step student outperforms the 1 024‑step teacher by 32 % perplexity while being 128× faster, beating basel…

    Apple Machine Learning Researchapple.com1 minpaper

  4. RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning

    RetireOPD introduces a self‑retiring on‑policy distillation framework for multi‑turn RL agents. A skill‑conditioned teacher is first trained with environment rewards, then a skill‑free student learns jointly via RL and token‑level distillation. The student automatically drops the teacher once its performance gap stops shrinking and it reaches a target success‑rate fraction, after which training c…

    Hugging Face Daily Papersarxiv.org1 minpaper

  5. When2Think: Learning Difficulty-Aware Length Control for Efficient Hybrid Reasoning Models

    When2Think introduces a post‑training framework that lets a large reasoning model decide per‑instance how much reasoning depth to allocate, using difficulty‑aware reward shaping (IDAC) and verifier rewards. It cuts token usage by ~28% while boosting Pass@3 by 10% on AIME24 and reaches 40% Pass@3 on AIME25, outperforming compression and routing baselines.

    Hugging Face Daily Papersarxiv.org1 minpaper

  6. Register Tokens for Bounded-State Reasoning in Diffusion Language Models

    Register tokens are fixed‑position embeddings that store a compact hidden state across diffusion‑based language model generation chunks, enabling bounded‑state reasoning without retaining all prior text. Post‑training on LLaDA and Dream shows up to +8.5 math and +19.5 code benchmark points versus plain text carry, and RL fine‑tuning further improves long‑horizon tasks.

    Hugging Face Daily Papersarxiv.org1 minpaper