proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersFanchao Chen, Ziheng Jiang, Ziyun Wei1 min readpaperadvanced

Towards Full Pipeline FP8 Reinforcement Learning for LLMs

Read on arxiv.org

Summary

FP8 quantization of RL pipelines for LLMs causes training instability because quantization noise skews importance ratios, zeroing gradients for negative‑advantage tokens and leading to entropy spikes. The authors introduce Calibrated Clipping, a dynamic adjustment of FP8 clipping bounds that matches BF16 quantile statistics, restoring stable training across GRPO and DAPO methods for 8‑32 B models…

  • FP8 quantization introduces bias in the importance ratio, pushing negative‑advantage tokens out of the trust region and nullifying their gradients.
  • Resulting instability manifests as mid‑training entropy surges and nonsensical model outputs.
  • Calibrated Clipping aligns FP8 clipping bounds with high‑precision BF16 distributions by matching the lower‑quantile and rebalancing the upper bound.
  • Experiments on GRPO and DAPO across 8B‑32B LLMs and various FP8 scaling granularities show the method eliminates entropy spikes and matches BF16 baselines.

Stable low‑precision training is critical for scaling RL‑based LLM fine‑tuning cost‑effectively. By fixing a previously hidden source of FP8 instability, the paper enables faster, cheaper RL training without sacrificing model quality.

8/10
llmmachine learningreinforcement learningperformancequantizationfp8

Related reading

  1. DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models

    DACA‑GRPO adds denoising‑aware credit assignment to GRPO‑style RL trainers for diffusion LLMs. It computes per‑token importance scores from intermediate denoising steps and uses stratified masking to reduce mean‑field bias in likelihood estimates. Plug‑and‑play on three existing GRPO methods, it yields consistent gains on seven downstream tasks (up to +5.6 pp math, +7.4 pp code, +36.3 pp constrai…

    Apple Machine Learning Researchapple.com1 minpaper

  2. Learning to solve hard problems in RL for LLMs by never giving up

    The post introduces the *Matthew Effect* in RL‑fine‑tuning of LLMs—performance gains concentrate on tasks the model already solves— and proposes *Never Give Up* (NGU), an adaptive sampling scheme that uses a small k for easy prompts and retries hard prompts with a high‑probability “never give up” loop. Experiments on math (AIME, GSM8k), code (Manufactoria), and larger‑scale setups (DeepScaler) sh…

    Hacker News front pagegithub.io11 minHN1179

  3. DataFlex-RL: An Evaluation Platform for RLVR Data Policies

    The paper introduces DataFlex‑RL, a platform to benchmark how different data‑selection policies affect reinforcement‑learning‑with‑verifiable‑rewards training. Across extensive experiments on Qwen2.5‑7B and Llama‑3.1‑8B, uniform sampling is the only method that consistently improves performance, and no alternative policy yields a statistically significant gain.

    Hugging Face Daily Papersarxiv.org1 minpaper

  6. Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening

    PPO critics in reinforcement learning for LLMs suffer from "Value Flattening," where predicted state values are too flat compared to actual values. This paper identifies the causes as an implicit variance penalty and redundant updates, and proposes SParse Proximal Policy Optimization (SP3O) to mitigate it by supervising only a few well-separated states.

    Hugging Face Daily Papersarxiv.org1 minpaper