Hacker News front pageMichael Noukhovitch11 min readadvanced
Learning to solve hard problems in RL for LLMs by never giving up
Summary
The post introduces the *Matthew Effect* in RL‑fine‑tuning of LLMs—performance gains concentrate on tasks the model already solves— and proposes *Never Give Up* (NGU), an adaptive sampling scheme that uses a small k for easy prompts and retries hard prompts with a high‑probability “never give up” loop. Experiments on math (AIME, GSM8k), code (Manufactoria), and larger‑scale setups (DeepScaler) sh…
- RL fine‑tuning of LLMs exhibits a *Matthew Effect*: improvements scale with the model’s initial competence, leaving hard problems largely untouched.
- Increasing the number of sampled completions (k) helps early on but later wastes compute on easy prompts that receive spurious incorrect completions.
- Never Give Up (NGU) adaptively samples: use a small k, filter solved prompts quickly, and with probability p re‑sample unsolved prompts, effectively creating a geometric‑distributed number of attempts.
- NGU with k=4 and p=0.9 matches or exceeds the performance of any fixed‑k GRPO baseline on GSM8k Platinum, especially on the hardest difficulty bucket.
RL‑based alignment and capability‑boosting for LLMs is increasingly used in production systems. If fine‑tuning only amplifies existing strengths, models will fail on the hardest, most safety‑critical tasks. NGU offers a simple, compute‑efficient way to redistribute training signal toward those hard…
7/10
