Search posts, papers, and topics
Pivot-SD is an efficient self-distillation framework for masked diffusion language models (dLMs) that addresses their credit-assignment problem by training only on high-impact "pivot" tokens. It improves LLaDA-8B-Instruct performance on math and code benchmarks using minimal data compared to other methods.
On-policy distillation (OPD) faces a challenge where the teacher's performance degrades when supervising student-generated, off-policy trajectories. The SCOUT framework addresses this by co-training the teacher to adapt to student-generated prefixes using reinforcement learning, consistently improving OPD effectiveness.
This paper introduces SBERT2S1, a method to convert biomedical sentence encoders into typed decision models, and BIODECIDE, a new biomedical typed-decision suite. It finds that retrieval training benefits prior-fused residual (PFR) models, cross-head (C) models generally outperform PFR, and the RLCD objective trails cross-entropy due to reward normalization issues.
