Hugging Face Daily PapersJianman Lin, Shailesh Shailesh, Zhongyi LuoSep 11, 20261 min readpaperadvanced

Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models

Summary

Latent Interface Training (LIT) first learns a goal‑conditioned action prior without visual input, then adds a pose‑supervised latent interface as the only visual conditioning path. Applied to several vision‑language‑action models, LIT cuts vision‑action shortcuts and lifts LIBERO‑Plus success by 3.9–10.7 points and real‑world task success by 13.3–16.7 points under distribution shifts.

Stage 1 trains an action expert on language, robot state, and terminal SE(3) pose, removing visual cues from goal‑directed action generation.
Stage 2 introduces a latent interface that fuses visual and semantic embeddings and is supervised to reconstruct the terminal pose, preserving task‑relevant spatial info.
Across four architectures, LIT improves LIBERO‑Plus success by 3.9–10.7 pp and real‑world success by 13.3–16.7 pp under unseen cameras, lighting, and distractors.
The approach is framework‑agnostic and can be retrofitted to existing vision‑language‑action models without architectural changes.

Robotics engineers and researchers building foundation models need a principled way to prevent vision‑action shortcuts that degrade performance under visual distribution shifts.

7/10

machine learning vision language robotics foundation models

Related reading

In-Context Robot Learning with VLM Agents arxiv.org
GPT‑Policy is a framework that lets a large vision‑language model (e.g. GPT‑6 Astra) perform in‑context robot learning: a context compiler extracts visual transitions from demos, the VLM proposes tool actions, and a constrained controller verifies and executes them. Real‑robot experiments show that raw video demos improve success rates even without explicit action labels, and that providing align…
Hugging Face Daily Papersarxiv.orgDongzhou Cheng, Taoran Yi, Ye Fang6d1 minpaper
LLaDA-UI: Bringing Block-wise Diffusion to Vision-Language GUI Agents arxiv.org
LLaDA‑UI is a 16.7 B‑parameter Mixture‑of‑Experts vision‑language model that uses block‑wise diffusion to generate GUI actions in parallel. It achieves state‑of‑the‑art performance on multiple GUI grounding and navigation benchmarks, beating larger Qwen models while keeping decoding latency low.
Hugging Face Daily Papersarxiv.orgZhangxuan Gu, Haoxing Chen, Qi Qin13d1 minpaper
ActionPiece: Rethinking Action Tokenization for Autoregressive Vision-Language-Action Models arxiv.org
Action tokenizers for autoregressive VLA models often fail to preserve subtle action adjustments, despite good pointwise reconstruction. This paper introduces Physical Rank Consistency (PRC) to measure relational fidelity and ActionPiece, a new tokenizer that uses joint supervision to preserve these physical relationships, significantly improving policy success on robotics benchmarks.
Hugging Face Daily Papersarxiv.orgShijie Lian, Bin Yu, Zhaolong Shen6d1 minpaper
HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harness arxiv.org
HarnessVLN introduces a zero‑shot, training‑free embodied navigation framework that wraps a multimodal LLM in an "Agent Harness" – a tool‑based protocol that validates planner actions against spatial evidence, tracks progress with hierarchical event memory, and maintains a persistent spatiotemporal graph for recovery. The system works for instruction‑following and object‑goal tasks, achieving 60.…
Hugging Face Daily Papersarxiv.orgYang Chen, Lirong Che, Zhenyu Huang8d1 minpaper
Learning Foresight without Explicit Trajectories for 3D Diffusion Policies arxiv.org
This paper introduces Movement Trend Guidance (MTG), a method to provide foresight to 3D diffusion policies for robotic manipulation without explicit trajectory planning. MTG learns a compact latent representation of interaction evolution, significantly improving performance on various benchmarks with minimal parameter overhead.
Hugging Face Daily Papersarxiv.orgZhongbo Zhang, Zaibin Zhang, Yifan Wang5d1 minpaper
From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention arxiv.org
The paper presents PARTS, a framework that augments a frozen pretrained robot policy with RL‑learned residuals on selected bottleneck subtasks, using local success rewards and minimal human resets. In real‑world bimanual and single‑arm tasks, PARTS more than doubles success rates with only minutes of robot rollouts, outperforming prior fine‑tuning methods.
Hugging Face Daily Papersarxiv.orgSichang Su, Benjamin Yang, Zhiyun Deng4d1 minpaper

Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models

Summary

Related reading

In-Context Robot Learning with VLM Agentsarxiv.org

LLaDA-UI: Bringing Block-wise Diffusion to Vision-Language GUI Agentsarxiv.org

ActionPiece: Rethinking Action Tokenization for Autoregressive Vision-Language-Action Modelsarxiv.org

HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harnessarxiv.org

Learning Foresight without Explicit Trajectories for 3D Diffusion Policiesarxiv.org

From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Interventionarxiv.org

In-Context Robot Learning with VLM Agents arxiv.org

LLaDA-UI: Bringing Block-wise Diffusion to Vision-Language GUI Agents arxiv.org

ActionPiece: Rethinking Action Tokenization for Autoregressive Vision-Language-Action Models arxiv.org

HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harness arxiv.org

Learning Foresight without Explicit Trajectories for 3D Diffusion Policies arxiv.org

From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention arxiv.org