proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersZhangxuan Gu, Haoxing Chen, Qi Qin1 min readpaperadvanced

LLaDA-UI: Bringing Block-wise Diffusion to Vision-Language GUI Agents

Read on arxiv.org

Summary

LLaDA‑UI is a 16.7 B‑parameter Mixture‑of‑Experts vision‑language model that uses block‑wise diffusion to generate GUI actions in parallel. It achieves state‑of‑the‑art performance on multiple GUI grounding and navigation benchmarks, beating larger Qwen models while keeping decoding latency low.

  • Block‑wise diffusion lets the language model generate multiple token blocks in parallel, cutting decoding latency for real‑time GUI control.
  • LLaDA‑UI combines a 16.7 B parameter Mixture‑of‑Experts backbone with a native‑resolution vision encoder, trained first on generic multimodal data then fine‑tuned on diverse GUI interaction logs.
  • On standard grounding and navigation benchmarks across mobile, desktop and web, LLaDA‑UI outperforms Qwen2.5‑VL‑7B and beats Qwen3‑VL‑8B on four of six GUI tasks.
  • The two‑stage pipeline shows that diffusion LLMs can be extended to multimodal agents without losing their parallel decoding advantage.

Engineers building UI automation or assistive agents need low‑latency multimodal models; block‑wise diffusion offers a concrete path to that.

7/10

Related reading

  1. llm-keys-ui 0.1

    Simon Willison released llm-keys-ui 0.1, a tiny web UI that lets you store LLM API keys on a remote machine and retrieve them via the llm CLI, avoiding pasting keys into chat sessions. It runs with a single uvx command and exposes local network URLs for access.

    Simon Willisonsimonwillison.net1 minrelease

  2. MintAct: A Unified Visual Agent for Digital Environments

    MintAct is a family of vision-language models (2B-8B) that unifies UI grounding, multi-step navigation, and visual tool use across mobile, desktop, and web environments. It achieves state-of-the-art performance by leveraging a scalable environment and an asynchronous reinforcement learning infrastructure.

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models

    Latent Interface Training (LIT) first learns a goal‑conditioned action prior without visual input, then adds a pose‑supervised latent interface as the only visual conditioning path. Applied to several vision‑language‑action models, LIT cuts vision‑action shortcuts and lifts LIBERO‑Plus success by 3.9–10.7 points and real‑world task success by 13.3–16.7 points under distribution shifts.

    Hugging Face Daily Papersarxiv.org1 minpaper

  4. Register Tokens for Bounded-State Reasoning in Diffusion Language Models

    Register tokens are fixed‑position embeddings that store a compact hidden state across diffusion‑based language model generation chunks, enabling bounded‑state reasoning without retaining all prior text. Post‑training on LLaDA and Dream shows up to +8.5 math and +19.5 code benchmark points versus plain text carry, and RL fine‑tuning further improves long‑horizon tasks.

    Hugging Face Daily Papersarxiv.org1 minpaper

  5. GLM 5.3 FlashX now available on AI Gateway

    Vercel AI Gateway now offers the GLM‑5.3‑FlashX model, a fast (~200 tps) multimodal coding LLM. The post includes a TypeScript streaming example, notes the model’s fit for coding agents and interactive tools, and lists AI Gateway features (unified API, usage tracking, retries/failover, custom reporting, key budgets, routing rules) with no platform fee.

    Vercelvercel.com1 minrelease

  6. Dynamically Scaled Activation Steering

    Dynamically Scaled Activation Steering (DSAS) is a method‑agnostic framework that learns per‑token, per‑layer scaling factors to turn existing activation‑steering interventions on only when a model is likely to produce undesired output (e.g., toxic text). The scaling can be optimized jointly with any steering function, improves the toxicity‑utility trade‑off on language models, transfers to text‑…

    Apple Machine Learning Researchapple.com1 minpaper