proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersHao Liang, Mingrui Chen, Hengyi Feng1 min readpaperadvanced

DataFlex-RL: An Evaluation Platform for RLVR Data Policies

Read on arxiv.org

Summary

The paper introduces DataFlex‑RL, a platform to benchmark how different data‑selection policies affect reinforcement‑learning‑with‑verifiable‑rewards training. Across extensive experiments on Qwen2.5‑7B and Llama‑3.1‑8B, uniform sampling is the only method that consistently improves performance, and no alternative policy yields a statistically significant gain.

  • Uniform rollout sampling (GRPO) yields a 7.76% absolute gain over an untrained checkpoint on a 12‑benchmark suite, and no alternative policy shows statistically significant improvement.
  • Across 13 data‑policy configurations, neither sophisticated rollout‑selection/reweighting methods nor adaptive mixtures outperform simple uniform sampling at 95% confidence.
  • Extending experiments to Llama‑3.1‑8B‑Base reproduces the same lack of clear winner, indicating results are not model‑specific.
  • Benchmark composition heavily influences rankings: a math‑heavy 6‑benchmark summary correlates –0.33 with the full 12‑benchmark ranking, highlighting evaluation sensitivity.

RL researchers and LLM fine‑tuning engineers should care because it shows that sophisticated data‑policy tricks may not outperform simple uniform sampling, saving engineering effort.

7/10

Related reading

  1. RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning

    RetireOPD introduces a self‑retiring on‑policy distillation framework for multi‑turn RL agents. A skill‑conditioned teacher is first trained with environment rewards, then a skill‑free student learns jointly via RL and token‑level distillation. The student automatically drops the teacher once its performance gap stops shrinking and it reaches a target success‑rate fraction, after which training c…

    Hugging Face Daily Papersarxiv.org1 minpaper

  2. Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening

    PPO critics in reinforcement learning for LLMs suffer from "Value Flattening," where predicted state values are too flat compared to actual values. This paper identifies the causes as an implicit variance penalty and redundant updates, and proposes SParse Proximal Policy Optimization (SP3O) to mitigate it by supervising only a few well-separated states.

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. In-Context Robot Learning with VLM Agents

    GPT‑Policy is a framework that lets a large vision‑language model (e.g. GPT‑6 Astra) perform in‑context robot learning: a context compiler extracts visual transitions from demos, the VLM proposes tool actions, and a constrained controller verifies and executes them. Real‑robot experiments show that raw video demos improve success rates even without explicit action labels, and that providing align…

    Hugging Face Daily Papersarxiv.org1 minpaper

  4. From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention

    The paper presents PARTS, a framework that augments a frozen pretrained robot policy with RL‑learned residuals on selected bottleneck subtasks, using local success rewards and minimal human resets. In real‑world bimanual and single‑arm tasks, PARTS more than doubles success rates with only minutes of robot rollouts, outperforming prior fine‑tuning methods.

    Hugging Face Daily Papersarxiv.org1 minpaper

  5. RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents

    The paper presents RecreationWorld, a five‑platform framework that lets hybrid computer‑use agents learn by recreating the behavior of a running reference, and introduces RecreationBench, a 250‑task benchmark with programmatic and visual assertions. Experiments show GPT‑6 Astra reaches 58.1% overall but struggles with deeper programmatic tests, highlighting gaps in current agents.

    Hugging Face Daily Papersarxiv.org2 minpaper