proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersFei Yuan, Changjiang Gao, Yilei Tu1 min readpaperadvanced

Drift-Constrained Optimization: Only Direction Matters in Fine-Tuning Instruct Models

Read on arxiv.org

Summary

The paper reframes fine‑tuning of instruction‑tuned LLMs as a direction‑selection problem under a fixed behavioral‑drift budget, showing that the update direction, not magnitude, determines trade‑offs between target performance and capability preservation. In QA‑only fine‑tuning of Qwen‑3 models, layer‑selective probing finds effective directions that boost scientific reasoning and multilingual t…

  • Drift is treated as a budgeted distance from the reference model; fine‑tuning then optimizes only the update direction.
  • A coarse layer‑selective probe can identify directions that improve a target task without degrading multi‑step reasoning.
  • On Qwen‑3‑8B/14B, direction‑constrained QA‑only fine‑tuning yields large gains in scientific reasoning and translation across 100+ languages, matching dedicated systems.
  • The resulting models serve as stronger initializations for subsequent RLHF pipelines.

LLM engineers and researchers who fine‑tune instruction models need a principled way to improve task performance while preserving existing capabilities.

8/10
llmoptimizationfine tuningmultilingualtranslationscientific reasoning

Related reading

  1. Training Specialist Models without Reasoning Trajectories for Domain Expert Distillation

    The paper shows that when specialist LLMs are trained only on QA pairs (no explicit reasoning supervision), their optimization implicitly selects a latent distribution of reasoning trajectories. By treating the distilled student as an agnostic probe—since it inherits only the sampled trajectories—the authors empirically demonstrate a strong correlation (across 27 specialist‑student pairs) between…

    Hugging Face Daily Papersarxiv.org1 minpaper

  2. Decoy Direction Optimization: A Post-Hoc Defense Against LLM Abliteration

    Decoy Direction Optimization (DDO) is a post‑hoc weight‑editing defense for open‑weight LLMs that injects a high‑magnitude nonlinear decoy into MLP neurons, corrupting contrastive estimators used by Refusal Feature Ablation (RFA) attacks. The paper proves a spectral bound on the effect, evaluates DDO on six model families (including Llama‑3‑8B‑Instruct), and shows <10 % attack success rate (ASR)…

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. Learning to solve hard problems in RL for LLMs by never giving up

    The post introduces the *Matthew Effect* in RL‑fine‑tuning of LLMs—performance gains concentrate on tasks the model already solves— and proposes *Never Give Up* (NGU), an adaptive sampling scheme that uses a small k for easy prompts and retries hard prompts with a high‑probability “never give up” loop. Experiments on math (AIME, GSM8k), code (Manufactoria), and larger‑scale setups (DeepScaler) sh…

    Hacker News front pagegithub.io11 minHN1179

  4. Why I'm still bearish on LLMs after Navier-Stokes

    The author argues that despite headline successes (e.g., Navier‑Stokes proof, security exploits), current frontier LLMs still require heavy human oversight and rigorous specifications that are costly to produce. Reward‑hacking, narrow generalization, and the need for domain‑expert spec writing limit autonomous deployment to only a few niche domains (high‑failure‑cost work, tightly defined tasks,…

    Hacker News front pagedank.systems5 minHN487644lobste.rs49

  5. A Zeroth-Order Paradigm for LLM Preference Alignment

    The paper proposes Comparison-based Preference Optimization (ComPO), a zeroth‑order method that uses comparison oracles to align LLMs without a differentiable loss. Experiments on several LLM families show it improves win rates and mitigates likelihood displacement compared to direct alignment approaches.

    Hugging Face Daily Papersarxiv.org1 minpaper

  6. How Value Induction Reshapes LLM Behaviour

    Apple researchers fine‑tune LLMs on curated subsets of value‑oriented preference data and measure cross‑value effects, safety, and anthropomorphic language. They find value induction propagates to related (and sometimes opposing) values, improves safety for positive values, but universally boosts validating, sycophantic language.

    Apple Machine Learning Researchapple.com1 minpaper