proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersYizhuo Li, Jianhao Yan, Yun Luo1 min readpaperadvanced

Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening

Read on arxiv.org

Summary

PPO critics in reinforcement learning for LLMs suffer from "Value Flattening," where predicted state values are too flat compared to actual values. This paper identifies the causes as an implicit variance penalty and redundant updates, and proposes SParse Proximal Policy Optimization (SP3O) to mitigate it by supervising only a few well-separated states.

  • PPO critics exhibit "Value Flattening," where predicted state values are overly flat despite sharp changes in true state values.
  • This failure mode is more pronounced in larger state spaces, impacting LLM reinforcement learning.
  • Value Flattening is linked to an implicit variance penalty in the critic loss and redundant updates from correlated states.
  • SP3O mitigates Value Flattening by applying the value loss to only a few well-separated states per response.

ML engineers working on reinforcement learning for large language models should care, as this paper identifies and addresses a critical, overlooked failure mode in PPO critic training.

8/10
llmmachine learningreinforcement learningdeep learningppo

Related reading

  1. DataFlex-RL: An Evaluation Platform for RLVR Data Policies

    The paper introduces DataFlex‑RL, a platform to benchmark how different data‑selection policies affect reinforcement‑learning‑with‑verifiable‑rewards training. Across extensive experiments on Qwen2.5‑7B and Llama‑3.1‑8B, uniform sampling is the only method that consistently improves performance, and no alternative policy yields a statistically significant gain.

    Hugging Face Daily Papersarxiv.org1 minpaper

  2. PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization

    PLC‑DPO extends Direct Preference Optimization by using the policy‑reference margin to route each training pair into clean, flipped, or tie categories, actively correcting noisy or ambiguous labels. Across extensive benchmarks it improves mean win‑rate from 55.5 % to 60.5 % and stays stable under injected noise and tie stress tests.

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning

    RetireOPD introduces a self‑retiring on‑policy distillation framework for multi‑turn RL agents. A skill‑conditioned teacher is first trained with environment rewards, then a skill‑free student learns jointly via RL and token‑level distillation. The student automatically drops the teacher once its performance gap stops shrinking and it reaches a target success‑rate fraction, after which training c…

    Hugging Face Daily Papersarxiv.org1 minpaper

  4. Generalized Agent Iteration: One Formal Framework for Iterative Policy Improvement and Recursive Self-Improvement

    The paper introduces Generalized Agent Iteration (GAI), a formal framework that unifies classical iterative policy improvement (GPI) and recursive self‑improvement (RSI). GAI treats an agent as a set of modifiable components and models learning as a loop of evaluation and improvement. Two binary “dials”—whether the improvement mechanism is internal to the agent and whether the evaluation standard…

    Hugging Face Daily Papersarxiv.org1 minpaper

  5. DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models

    DACA‑GRPO adds denoising‑aware credit assignment to GRPO‑style RL trainers for diffusion LLMs. It computes per‑token importance scores from intermediate denoising steps and uses stratified masking to reduce mean‑field bias in likelihood estimates. Plug‑and‑play on three existing GRPO methods, it yields consistent gains on seven downstream tasks (up to +5.6 pp math, +7.4 pp code, +36.3 pp constrai…

    Apple Machine Learning Researchapple.com1 minpaper

  6. Online Learning with LLM Experts from Limited Feedback

    The paper models prompt routing to multiple LLM experts as a bandit problem with limited feedback and proposes algorithms that achieve sublinear regret in both full‑information and bandit settings. Experiments demonstrate that the methods learn effective routing strategies across diverse LLMs using only a small feedback budget.

    Hugging Face Daily Papersarxiv.org2 minpaper