Hugging Face Daily PapersDongzhou Cheng, Taoran Yi, Ye FangSep 16, 20261 min readpaperadvanced

In-Context Robot Learning with VLM Agents

Summary

GPT‑Policy is a framework that lets a large vision‑language model (e.g. GPT‑6 Astra) perform in‑context robot learning: a context compiler extracts visual transitions from demos, the VLM proposes tool actions, and a constrained controller verifies and executes them. Real‑robot experiments show that raw video demos improve success rates even without explicit action labels, and that providing align…

In‑context learning can be operationalized for robots by feeding VLMs a compact visual‑transition context rather than fine‑tuning parameters.
A verification controller is essential to filter VLM‑proposed actions before execution, improving safety and reliability.
Human video demonstrations alone can boost task completion, suggesting that VLMs can infer implicit action semantics.
Providing explicit action references (aligned demonstrations) yields measurable gains on tasks requiring precise contact.

If robots can adapt on‑the‑fly from a few demonstrations without retraining, deployment in unstructured environments becomes far more feasible. This work shows a concrete path to leverage the broad reasoning abilities of commercial VLMs for embodied tasks, bridging the gap between language‑level un…

5/10

robotics ml infra vision language models in context learning embodied ai

Related reading

From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention arxiv.org
The paper presents PARTS, a framework that augments a frozen pretrained robot policy with RL‑learned residuals on selected bottleneck subtasks, using local success rewards and minimal human resets. In real‑world bimanual and single‑arm tasks, PARTS more than doubles success rates with only minutes of robot rollouts, outperforming prior fine‑tuning methods.
Hugging Face Daily Papersarxiv.orgSichang Su, Benjamin Yang, Zhiyun Deng4d1 minpaper
DataFlex-RL: An Evaluation Platform for RLVR Data Policies arxiv.org
The paper introduces DataFlex‑RL, a platform to benchmark how different data‑selection policies affect reinforcement‑learning‑with‑verifiable‑rewards training. Across extensive experiments on Qwen2.5‑7B and Llama‑3.1‑8B, uniform sampling is the only method that consistently improves performance, and no alternative policy yields a statistically significant gain.
Hugging Face Daily Papersarxiv.orgHao Liang, Mingrui Chen, Hengyi Feng17d1 minpaper
Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models arxiv.org
Latent Interface Training (LIT) first learns a goal‑conditioned action prior without visual input, then adds a pose‑supervised latent interface as the only visual conditioning path. Applied to several vision‑language‑action models, LIT cuts vision‑action shortcuts and lifts LIBERO‑Plus success by 3.9–10.7 points and real‑world task success by 13.3–16.7 points under distribution shifts.
Hugging Face Daily Papersarxiv.orgJianman Lin, Shailesh Shailesh, Zhongyi Luo11d1 minpaper
ActionPiece: Rethinking Action Tokenization for Autoregressive Vision-Language-Action Models arxiv.org
Action tokenizers for autoregressive VLA models often fail to preserve subtle action adjustments, despite good pointwise reconstruction. This paper introduces Physical Rank Consistency (PRC) to measure relational fidelity and ActionPiece, a new tokenizer that uses joint supervision to preserve these physical relationships, significantly improving policy success on robotics benchmarks.
Hugging Face Daily Papersarxiv.orgShijie Lian, Bin Yu, Zhaolong Shen6d1 minpaper
GAVEL: Graph World Models for Verified and Efficient Long-Horizon LLM Task Planning arxiv.org
GAVEL augments LLM‑driven robot planners with an explicit graph world model that verifies actions, repairs violations, and reasons over belief distributions, boosting single‑task success from 41 % to 92 % and multi‑task success from 20 % to 93 % on BEHAVIOR‑1K.
Hugging Face Daily Papersarxiv.orgRuiyang Wang, Hao-Lun Hsu, Swarajh Mehta6d1 minpaper
RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning arxiv.org
RetireOPD introduces a self‑retiring on‑policy distillation framework for multi‑turn RL agents. A skill‑conditioned teacher is first trained with environment rewards, then a skill‑free student learns jointly via RL and token‑level distillation. The student automatically drops the teacher once its performance gap stops shrinking and it reaches a target success‑rate fraction, after which training c…
Hugging Face Daily Papersarxiv.orgYan Yu, Zhengxi Lu, Yizhou Liu5d1 minpaper

In-Context Robot Learning with VLM Agents

Summary

Related reading

From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Interventionarxiv.org

DataFlex-RL: An Evaluation Platform for RLVR Data Policiesarxiv.org

Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Modelsarxiv.org

ActionPiece: Rethinking Action Tokenization for Autoregressive Vision-Language-Action Modelsarxiv.org

GAVEL: Graph World Models for Verified and Efficient Long-Horizon LLM Task Planningarxiv.org

RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learningarxiv.org

From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention arxiv.org

DataFlex-RL: An Evaluation Platform for RLVR Data Policies arxiv.org

Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models arxiv.org

ActionPiece: Rethinking Action Tokenization for Autoregressive Vision-Language-Action Models arxiv.org

GAVEL: Graph World Models for Verified and Efficient Long-Horizon LLM Task Planning arxiv.org

RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning arxiv.org