Hugging Face Daily PapersNaveen Vakada, Mingyuan Li, Shaoxiong JiSep 28, 20261 min readpaperadvanced

Label-free steering: Compressing test-time reinforcement learning into bias-only subspaces

Summary

This paper introduces label-free bias-only Test-Time Reinforcement Learning (TTRL), which uses majority-vote pseudolabels and optimizes only ~100K bias parameters. It achieves 76.67% accuracy on MATH-500 with Qwen2.5-7B, optimizing 76,000x fewer parameters than full-parameter TTRL, demonstrating substantial adaptation from a tiny subspace.

TTRL can be effective by optimizing only ~100K bias parameters, keeping the pretrained backbone frozen.
Majority-vote pseudolabels provide sufficient reward signals for label-free test-time adaptation.
The method achieved 76.67% accuracy on MATH-500 with Qwen2.5-7B, comparable to labeled bias-steering.
It optimizes 76,000x fewer parameters than full-parameter TTRL, a significant efficiency gain.

This work is significant for LLM researchers and practitioners seeking highly efficient, label-free methods for improving model reasoning and adaptation at inference time.

8/10

llm machine learning reinforcement learning efficiency fine tuning model optimization

Related reading

Harness-Zero: Harness Distillation via Agent-as-Harness arxiv.org
Harness-Zero proposes a harness‑distillation technique where a specialized harness guides a student model via an intermediate agent‑as‑harness, allowing the learned behavior to be baked into the model weights and removed at deployment. Experiments show macro‑average task success jumps from 23.3% to 44.3%, surpassing the 41.7% achieved with the harness still attached, and recovers 82.3% of harness…
Hugging Face Daily Papersarxiv.orgHaoran Ye, Yuxing Lu, Haonan Dong14d1 minpaper
WISE-ATTA: When to Ask for Labels in Budgeted Active Test-Time Adaptation arxiv.org
WISE-ATTA addresses budgeted active test-time adaptation (ATTA), where labels are only available for a fraction of test batches. It proposes a budget-aware strategy to decide when to request supervision over time, prioritizing useful periods and using drift-based sample selection.
Hugging Face Daily Papersarxiv.orgMuhammad Huzaifa, Lea Schönherr, Thorsten Eisenhofer6d1 minpaper
RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning arxiv.org
RetireOPD introduces a self‑retiring on‑policy distillation framework for multi‑turn RL agents. A skill‑conditioned teacher is first trained with environment rewards, then a skill‑free student learns jointly via RL and token‑level distillation. The student automatically drops the teacher once its performance gap stops shrinking and it reaches a target success‑rate fraction, after which training c…
Hugging Face Daily Papersarxiv.orgYan Yu, Zhengxi Lu, Yizhou Liu18d1 minpaper
DataFlex-RL: An Evaluation Platform for RLVR Data Policies arxiv.org
The paper introduces DataFlex‑RL, a platform to benchmark how different data‑selection policies affect reinforcement‑learning‑with‑verifiable‑rewards training. Across extensive experiments on Qwen2.5‑7B and Llama‑3.1‑8B, uniform sampling is the only method that consistently improves performance, and no alternative policy yields a statistically significant gain.
Hugging Face Daily Papersarxiv.orgHao Liang, Mingrui Chen, Hengyi Feng1mo1 minpaper
PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization arxiv.org
PLC‑DPO extends Direct Preference Optimization by using the policy‑reference margin to route each training pair into clean, flipped, or tie categories, actively correcting noisy or ambiguous labels. Across extensive benchmarks it improves mean win‑rate from 55.5 % to 60.5 % and stays stable under injected noise and tie stress tests.
Hugging Face Daily Papersarxiv.orgBoryeong Cho, Sumyeong Ahn, Se-Young Yun1mo1 minpaper
RRSI: Regularized Recursive Self-Improvement of Agent Harnesses arxiv.org
RRSI adds budget‑annealed edit limits, trajectory exploration incentives, a benchmark‑aware critic, and a pruning step to recursive self‑improvement of LLM agent harnesses, yielding up to +14.1 points on in‑distribution tasks and +4.7 on OOD benchmarks while cutting policy token usage by 30%.
Hugging Face Daily Papersarxiv.orgPeng Xia, Rujun Han, Zifeng Wang14d1 minpaperHN1

Label-free steering: Compressing test-time reinforcement learning into bias-only subspaces

Summary

Related reading

Harness-Zero: Harness Distillation via Agent-as-Harnessarxiv.org

WISE-ATTA: When to Ask for Labels in Budgeted Active Test-Time Adaptationarxiv.org

RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learningarxiv.org

DataFlex-RL: An Evaluation Platform for RLVR Data Policiesarxiv.org

PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimizationarxiv.org

RRSI: Regularized Recursive Self-Improvement of Agent Harnessesarxiv.org

Harness-Zero: Harness Distillation via Agent-as-Harness arxiv.org

WISE-ATTA: When to Ask for Labels in Budgeted Active Test-Time Adaptation arxiv.org

RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning arxiv.org

DataFlex-RL: An Evaluation Platform for RLVR Data Policies arxiv.org

PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization arxiv.org

RRSI: Regularized Recursive Self-Improvement of Agent Harnesses arxiv.org