proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersPeter Chen, Xi Chen, Wotao Yin1 min readpaperadvanced

A Zeroth-Order Paradigm for LLM Preference Alignment

Read on arxiv.org

Summary

The paper proposes Comparison-based Preference Optimization (ComPO), a zeroth‑order method that uses comparison oracles to align LLMs without a differentiable loss. Experiments on several LLM families show it improves win rates and mitigates likelihood displacement compared to direct alignment approaches.

  • ComPO extracts directional information from preference pairs via comparison oracles, avoiding direct optimization of a differentiable loss.
  • The offline version of ComPO has a convergence guarantee under smoothness, gradient sparsity, and oracle‑objective compatibility assumptions.
  • Online ComPO adds reverse‑KL control using unlabeled policy generations relative to a reference policy.
  • The authors prove performance bounds for a constrained scheme assuming local coverage and accurate in‑distribution pairwise rewards.

LLM alignment engineers and researchers should care because ComPO offers a theoretically grounded, potentially more efficient alternative to standard preference‑based fine‑tuning.

7/10
llmreinforcement learningoptimizationpreference alignment

Related reading

  1. PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization

    PLC‑DPO extends Direct Preference Optimization by using the policy‑reference margin to route each training pair into clean, flipped, or tie categories, actively correcting noisy or ambiguous labels. Across extensive benchmarks it improves mean win‑rate from 55.5 % to 60.5 % and stays stable under injected noise and tie stress tests.

    Hugging Face Daily Papersarxiv.org1 minpaper

  2. How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus

    This paper investigates the "lossless" claim of Orthrus, a hybrid architecture for accelerating LLM inference. It finds that under BF16 precision, Orthrus diverges from the exact autoregressive output trajectory in over 50% of cases, though FP32 maintains exact matching. Despite BF16 divergence, downstream task performance was not systematically degraded.

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. Drift-Constrained Optimization: Only Direction Matters in Fine-Tuning Instruct Models

    The paper reframes fine‑tuning of instruction‑tuned LLMs as a direction‑selection problem under a fixed behavioral‑drift budget, showing that the update direction, not magnitude, determines trade‑offs between target performance and capability preservation. In QA‑only fine‑tuning of Qwen‑3 models, layer‑selective probing finds effective directions that boost scientific reasoning and multilingual t…

    Hugging Face Daily Papersarxiv.org1 minpaper

  4. Learning to solve hard problems in RL for LLMs by never giving up

    The post introduces the *Matthew Effect* in RL‑fine‑tuning of LLMs—performance gains concentrate on tasks the model already solves— and proposes *Never Give Up* (NGU), an adaptive sampling scheme that uses a small k for easy prompts and retries hard prompts with a high‑probability “never give up” loop. Experiments on math (AIME, GSM8k), code (Manufactoria), and larger‑scale setups (DeepScaler) sh…

    Hacker News front pagegithub.io11 minHN1179

  5. How Value Induction Reshapes LLM Behaviour

    Apple researchers fine‑tune LLMs on curated subsets of value‑oriented preference data and measure cross‑value effects, safety, and anthropomorphic language. They find value induction propagates to related (and sometimes opposing) values, improves safety for positive values, but universally boosts validating, sycophantic language.

    Apple Machine Learning Researchapple.com1 minpaper

  6. Introducing System One Models and Jev

    TypeSafe AI announced its first “System One” model, Jev, a non‑text‑generating LLM that outputs type‑safe structured decisions with calibrated probabilities. It claims 40‑200× lower latency (70‑500 ms) and 100‑500× lower cost versus frontier LLMs, no hallucinations, and parallel sampling. The post includes a side‑by‑side demo, a custom “workflow” benchmark comparing Jev to GPT‑5.6/6 and other mod…

    Hacker News front pagetypesafe.ai9 minHN1824480lobste.rs26