proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersInjin Kong, Sunghwan Choi, Yohan Jo1 min readpaperadvanced

Unmask the State: When Does State Adaptation Matter for Masked Diffusion Language Models

Read on arxiv.org

Summary

This paper investigates when to adapt unmasking strategies in Masked Diffusion Language Models (MDMs) during inference. It finds that "adaptation opportunities" are highly heterogeneous and that selective adaptation, guided by lightweight detectors, is more effective than uniform adaptation.

  • MDM inference strategies (score, cardinality, region, commitment, planning) can be adapted during generation.
  • "Adaptation opportunity" quantifies the one-step utility gain from choosing a better action over a fixed one.
  • Adaptation opportunities are highly variable across different MDMs and tasks.
  • Selective adaptation, targeting high-opportunity states, significantly improves performance over fixed strategies.

Researchers and practitioners working with Masked Diffusion Language Models can use these insights to design more efficient and effective inference strategies, improving generation quality and resource utilization.

7/10
llmmachine learningoptimizationnlpinferencediffusion models

Related reading

  2. DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models

    DACA‑GRPO adds denoising‑aware credit assignment to GRPO‑style RL trainers for diffusion LLMs. It computes per‑token importance scores from intermediate denoising steps and uses stratified masking to reduce mean‑field bias in likelihood estimates. Plug‑and‑play on three existing GRPO methods, it yields consistent gains on seven downstream tasks (up to +5.6 pp math, +7.4 pp code, +36.3 pp constrai…

    Apple Machine Learning Researchapple.com1 minpaper

  3. The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trends

    The paper surveys recent attention variants in large language models, introducing a five‑dimensional framework (Memory Representation, Update, Access, Readout, Integration) to compare them. It shows that modern LLMs increasingly treat contextual memory as a coordinated, multi‑layer resource rather than a single attention operator.

    Hugging Face Daily Papersarxiv.org1 minpaper

  5. Register Tokens for Bounded-State Reasoning in Diffusion Language Models

    Register tokens are fixed‑position embeddings that store a compact hidden state across diffusion‑based language model generation chunks, enabling bounded‑state reasoning without retaining all prior text. Post‑training on LLaDA and Dream shows up to +8.5 math and +19.5 code benchmark points versus plain text carry, and RL fine‑tuning further improves long‑horizon tasks.

    Hugging Face Daily Papersarxiv.org1 minpaper

  6. A Lie Detector Test for Language Models: Reading Knowledge a Model Won't Reveal

    The paper presents Probe of Internal Recognition (PIR), a reference‑free technique that reads a language model’s internal activations to detect which answer it recognizes, achieving 70‑87% balanced accuracy across eight LLMs. PIR reliably distinguishes deliberate concealment from lack of knowledge, enabling audits of sandbagging and unlearning.

    Hugging Face Daily Papersarxiv.org1 minpaper