Hugging Face Daily PapersYan Yu, Zhengxi Lu, Yizhou LiuSep 17, 20261 min readpaperadvanced

RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning

Summary

RetireOPD introduces a self‑retiring on‑policy distillation framework for multi‑turn RL agents. A skill‑conditioned teacher is first trained with environment rewards, then a skill‑free student learns jointly via RL and token‑level distillation. The student automatically drops the teacher once its performance gap stops shrinking and it reaches a target success‑rate fraction, after which training c…

Decoupling teacher training (skill‑conditioned) from student training avoids unreliable privileged supervision.
Adaptive Retirement lets the student autonomously stop distillation when marginal benefit vanishes, simplifying schedule design.
Dense token‑level supervision from a teacher can substantially boost multi‑turn agent performance on benchmark tasks.

Dense supervision is a known bottleneck for RL agents that only receive sparse trajectory rewards. RetireOPD provides a practical, automated way to inject token‑level guidance without hand‑crafted schedules, and demonstrates that the student can surpass its teacher, suggesting the distilled knowled…

8/10

llm reinforcement learning distillation agentic language models

Related reading

Generalized Agent Iteration: One Formal Framework for Iterative Policy Improvement and Recursive Self-Improvement arxiv.org
The paper introduces Generalized Agent Iteration (GAI), a formal framework that unifies classical iterative policy improvement (GPI) and recursive self‑improvement (RSI). GAI treats an agent as a set of modifiable components and models learning as a loop of evaluation and improvement. Two binary “dials”—whether the improvement mechanism is internal to the agent and whether the evaluation standard…
Hugging Face Daily Papersarxiv.orgHongyao Tang, Yi Ma, Pengyi Li11d1 minpaper
Calibrating Teacher--Student Discrepancy for On-Policy Distillation arxiv.org
This paper introduces Calibrated On-Policy Distillation (Cal-OPD), a method to improve knowledge distillation by filtering out the teacher model's inherent deviations from the learning signal. Cal-OPD estimates and removes the teacher's self-deviation, leading to better student performance on mathematical reasoning tasks.
Hugging Face Daily Papersarxiv.orgQiangqiang He, Jin Li, MingCai Chen4d1 minpaper
DataFlex-RL: An Evaluation Platform for RLVR Data Policies arxiv.org
The paper introduces DataFlex‑RL, a platform to benchmark how different data‑selection policies affect reinforcement‑learning‑with‑verifiable‑rewards training. Across extensive experiments on Qwen2.5‑7B and Llama‑3.1‑8B, uniform sampling is the only method that consistently improves performance, and no alternative policy yields a statistically significant gain.
Hugging Face Daily Papersarxiv.orgHao Liang, Mingrui Chen, Hengyi Feng17d1 minpaper
In-Context Robot Learning with VLM Agents arxiv.org
GPT‑Policy is a framework that lets a large vision‑language model (e.g. GPT‑6 Astra) perform in‑context robot learning: a context compiler extracts visual transitions from demos, the VLM proposes tool actions, and a constrained controller verifies and executes them. Real‑robot experiments show that raw video demos improve success rates even without explicit action labels, and that providing align…
Hugging Face Daily Papersarxiv.orgDongzhou Cheng, Taoran Yi, Ye Fang6d1 minpaper
When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation arxiv.org
On-policy distillation (OPD) can lead to excessively long student responses, a phenomenon called length inflation. This paper identifies "termination-token mismatch" between base students and post-trained teachers as a key source, where models place stopping probability on different EOS tokens. Treating functionally equivalent EOS tokens as a shared semantic stopping action substantially mitigate…
Hugging Face Daily Papersarxiv.orgYuxiao Yang, Tianrun Yu, Shangzhe Li5d1 minpaper
EvolveTrade: Experience-Driven Policy Refinement for Self-Evolving LLM Trading Agents arxiv.org
EvolveTrade is a framework for self-evolving LLM trading agents that refines their tool-use policies (system prompts) based on past performance. It demonstrates improved Sharpe Ratio and Cumulative Return compared to agents with static policies across various market regimes.
Hugging Face Daily Papersarxiv.orgSehee Kim, Yumin Choi, Minki Kang7d1 minpaper

RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning

Summary

Related reading

Generalized Agent Iteration: One Formal Framework for Iterative Policy Improvement and Recursive Self-Improvementarxiv.org

Calibrating Teacher--Student Discrepancy for On-Policy Distillationarxiv.org

DataFlex-RL: An Evaluation Platform for RLVR Data Policiesarxiv.org

In-Context Robot Learning with VLM Agentsarxiv.org

When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillationarxiv.org

EvolveTrade: Experience-Driven Policy Refinement for Self-Evolving LLM Trading Agentsarxiv.org

Generalized Agent Iteration: One Formal Framework for Iterative Policy Improvement and Recursive Self-Improvement arxiv.org

Calibrating Teacher--Student Discrepancy for On-Policy Distillation arxiv.org

DataFlex-RL: An Evaluation Platform for RLVR Data Policies arxiv.org

In-Context Robot Learning with VLM Agents arxiv.org

When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation arxiv.org

EvolveTrade: Experience-Driven Policy Refinement for Self-Evolving LLM Trading Agents arxiv.org