Hugging Face Daily PapersMeng-Hao Guo, Zhe-Han Mo, Jia-Jun WangSep 19, 20261 min readpaperadvanced

Transferring the Intelligence of VLMs to Robotic Control

Summary

RoboDawn lets a vision‑language model (VLM) drive a robot via a tiny discrete command set (translate/rotate/gripper). Using a few in‑context demos, the VLM learns the interface and task strategy, then runs closed‑loop: observe image → reason → act → re‑observe. On the RoboTwin 2.0 C2R benchmark RoboDawn hits 53.2 % success zero‑shot, 73.6 % with one demo (vs. 46 % baseline). On RoboDojo it goes f…

A compact, human‑intuitive command interface can bridge VLM reasoning to low‑level robot actuation.
Few‑shot in‑context learning is enough to ground a VLM in both the interface and the task, yielding large performance jumps.
Zero‑shot VLM control already beats several policies trained on the same robot data, showing strong cross‑domain generalisation.
The approach transfers from simulation (RoboTwin, RoboDojo) to a physical Franka arm with minimal adaptation.

If VLMs can be repurposed for real‑world robot control with only a handful of demonstrations, we can leverage massive pre‑trained language‑vision models for new robotic tasks without costly data collection or reinforcement learning pipelines.

7/10

llm robotics vision language models in context learning simulation control

Related reading

In-Context Robot Learning with VLM Agents arxiv.org
GPT‑Policy is a framework that lets a large vision‑language model (e.g. GPT‑6 Astra) perform in‑context robot learning: a context compiler extracts visual transitions from demos, the VLM proposes tool actions, and a constrained controller verifies and executes them. Real‑robot experiments show that raw video demos improve success rates even without explicit action labels, and that providing align…
Hugging Face Daily Papersarxiv.orgDongzhou Cheng, Taoran Yi, Ye Fang6d1 minpaper
VABench: Measuring Embodied Spatial Intelligence through Visual Demonstrations, Active Perception, and Metric Control arxiv.org
VA‑Bench is a new benchmark that evaluates general‑purpose multimodal LLMs on the full observe‑reason‑act‑revise loop in embodied robotics, using RGB demonstrations, active camera control, and metric Cartesian commands. The best model reaches 53.9% average task success, showing active perception helps but long‑horizon tasks remain unsolved.
Hugging Face Daily Papersarxiv.orgZhongbo Zhang, Jiayi Jin, Yifan Wang5d1 minpaper
Think Like a World Model, Act Like a VLA: Distilling World-Model Representations into Compact Robot Policies arxiv.org
Distill frozen world‑model features into a Vision‑Language‑Action policy via a single alignment loss; no teacher at train time, no extra runtime cost. 0.8 B student runs 32 ms / 1.86 GB on RTX 5090, hits 97.9 % on LIBERO and improves RoboCasa‑GR1 from 48.2 % to 50.5 %, with transfer to real single‑arm and bimanual robots.
Hugging Face Daily Papersarxiv.orgTrung Dao, Sankalp Yamsani, Jaden Park1d1 minpaper
GAVEL: Graph World Models for Verified and Efficient Long-Horizon LLM Task Planning arxiv.org
GAVEL augments LLM‑driven robot planners with an explicit graph world model that verifies actions, repairs violations, and reasons over belief distributions, boosting single‑task success from 41 % to 92 % and multi‑task success from 20 % to 93 % on BEHAVIOR‑1K.
Hugging Face Daily Papersarxiv.orgRuiyang Wang, Hao-Lun Hsu, Swarajh Mehta6d1 minpaper
From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention arxiv.org
The paper presents PARTS, a framework that augments a frozen pretrained robot policy with RL‑learned residuals on selected bottleneck subtasks, using local success rewards and minimal human resets. In real‑world bimanual and single‑arm tasks, PARTS more than doubles success rates with only minutes of robot rollouts, outperforming prior fine‑tuning methods.
Hugging Face Daily Papersarxiv.orgSichang Su, Benjamin Yang, Zhiyun Deng4d1 minpaper
HuRo: Robotizing Human Videos for Scalable VLA Pretraining arxiv.org
The paper introduces a pipeline that converts heterogeneous human videos into robot‑aligned observations and actions, creating the 630K‑episode HuRo dataset. Pretraining vision‑language‑action (VLA) policies on this data boosts real‑world manipulation success from ~51% to ~80% and improves out‑of‑distribution robustness.
Hugging Face Daily Papersarxiv.orgJinho Jeong, Se June Joo, Jaehyun Kang4d1 minpaper

Transferring the Intelligence of VLMs to Robotic Control

Summary

Related reading

In-Context Robot Learning with VLM Agentsarxiv.org

VABench: Measuring Embodied Spatial Intelligence through Visual Demonstrations, Active Perception, and Metric Controlarxiv.org

Think Like a World Model, Act Like a VLA: Distilling World-Model Representations into Compact Robot Policiesarxiv.org

GAVEL: Graph World Models for Verified and Efficient Long-Horizon LLM Task Planningarxiv.org

From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Interventionarxiv.org

HuRo: Robotizing Human Videos for Scalable VLA Pretrainingarxiv.org

In-Context Robot Learning with VLM Agents arxiv.org

VABench: Measuring Embodied Spatial Intelligence through Visual Demonstrations, Active Perception, and Metric Control arxiv.org

Think Like a World Model, Act Like a VLA: Distilling World-Model Representations into Compact Robot Policies arxiv.org

GAVEL: Graph World Models for Verified and Efficient Long-Horizon LLM Task Planning arxiv.org

From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention arxiv.org

HuRo: Robotizing Human Videos for Scalable VLA Pretraining arxiv.org