proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersBoryeong Cho, Sumyeong Ahn, Se-Young Yun1 min readpaperadvanced

PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization

Read on arxiv.org

Summary

PLC‑DPO extends Direct Preference Optimization by using the policy‑reference margin to route each training pair into clean, flipped, or tie categories, actively correcting noisy or ambiguous labels. Across extensive benchmarks it improves mean win‑rate from 55.5 % to 60.5 % and stays stable under injected noise and tie stress tests.

  • PLC‑DPO routes each preference pair into clean, flipped, or tie cases using a calibrated policy‑reference margin, avoiding simple filtering of uncertain data.
  • It actively corrects supervision direction and strength, flipping or down‑weighting noisy labels during training.
  • On 57 dataset‑model‑benchmark cells PLC‑DPO achieves a mean win‑rate of 60.5 % versus 55.5 % for vanilla DPO, beating other baselines.
  • Injected‑noise and tie stress tests show the routing remains stable and correctly distinguishes flipped from weakly directional pairs.

Practitioners training LLMs with human feedback should care because noisy preference data can degrade alignment, and PLC‑DPO offers a drop‑in way to make the learning robust.

6/10
llmmachine learningpreference optimizationalignmentrobustness

Related reading

  1. A Zeroth-Order Paradigm for LLM Preference Alignment

    The paper proposes Comparison-based Preference Optimization (ComPO), a zeroth‑order method that uses comparison oracles to align LLMs without a differentiable loss. Experiments on several LLM families show it improves win rates and mitigates likelihood displacement compared to direct alignment approaches.

    Hugging Face Daily Papersarxiv.org1 minpaper

  2. DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models

    DACA‑GRPO adds denoising‑aware credit assignment to GRPO‑style RL trainers for diffusion LLMs. It computes per‑token importance scores from intermediate denoising steps and uses stratified masking to reduce mean‑field bias in likelihood estimates. Plug‑and‑play on three existing GRPO methods, it yields consistent gains on seven downstream tasks (up to +5.6 pp math, +7.4 pp code, +36.3 pp constrai…

    Apple Machine Learning Researchapple.com1 minpaper

  3. Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening

    PPO critics in reinforcement learning for LLMs suffer from "Value Flattening," where predicted state values are too flat compared to actual values. This paper identifies the causes as an implicit variance penalty and redundant updates, and proposes SParse Proximal Policy Optimization (SP3O) to mitigate it by supervising only a few well-separated states.

    Hugging Face Daily Papersarxiv.org1 minpaper

  4. DataFlex-RL: An Evaluation Platform for RLVR Data Policies

    The paper introduces DataFlex‑RL, a platform to benchmark how different data‑selection policies affect reinforcement‑learning‑with‑verifiable‑rewards training. Across extensive experiments on Qwen2.5‑7B and Llama‑3.1‑8B, uniform sampling is the only method that consistently improves performance, and no alternative policy yields a statistically significant gain.

    Hugging Face Daily Papersarxiv.org1 minpaper

  5. Learning to solve hard problems in RL for LLMs by never giving up

    The post introduces the *Matthew Effect* in RL‑fine‑tuning of LLMs—performance gains concentrate on tasks the model already solves— and proposes *Never Give Up* (NGU), an adaptive sampling scheme that uses a small k for easy prompts and retries hard prompts with a high‑probability “never give up” loop. Experiments on math (AIME, GSM8k), code (Manufactoria), and larger‑scale setups (DeepScaler) sh…

    Hacker News front pagegithub.io11 minHN1179

  6. Dynamically Scaled Activation Steering

    Dynamically Scaled Activation Steering (DSAS) is a method‑agnostic framework that learns per‑token, per‑layer scaling factors to turn existing activation‑steering interventions on only when a model is likely to produce undesired output (e.g., toxic text). The scaling can be optimized jointly with any steering function, improves the toxicity‑utility trade‑off on language models, transfers to text‑…

    Apple Machine Learning Researchapple.com1 minpaper