Hugging Face Daily PapersZhongbo Zhang, Jiayi Jin, Yifan WangSep 17, 20261 min readpaperadvanced

VABench: Measuring Embodied Spatial Intelligence through Visual Demonstrations, Active Perception, and Metric Control

Summary

VA‑Bench is a new benchmark that evaluates general‑purpose multimodal LLMs on the full observe‑reason‑act‑revise loop in embodied robotics, using RGB demonstrations, active camera control, and metric Cartesian commands. The best model reaches 53.9% average task success, showing active perception helps but long‑horizon tasks remain unsolved.

Active camera control more than doubles success on a matched task (27.86% → 57.50%).
Models achieve near‑perfect target localization (100%) but only ~79% on spatial relation inference.
General‑purpose MLLMs struggle with long‑horizon multi‑object compositions, completing none fully.
Held‑out geometry/layout variants can drop success by >30 points, highlighting limited transfer.

Anyone building embodied AI or robotic agents needs a realistic, open‑source benchmark to measure active perception and metric control capabilities of LLM‑based policies.

7/10

llm computer vision robotics benchmark embodied ai

Related reading

Transferring the Intelligence of VLMs to Robotic Control arxiv.org
RoboDawn lets a vision‑language model (VLM) drive a robot via a tiny discrete command set (translate/rotate/gripper). Using a few in‑context demos, the VLM learns the interface and task strategy, then runs closed‑loop: observe image → reason → act → re‑observe. On the RoboTwin 2.0 C2R benchmark RoboDawn hits 53.2 % success zero‑shot, 73.6 % with one demo (vs. 46 % baseline). On RoboDojo it goes f…
Hugging Face Daily Papersarxiv.orgMeng-Hao Guo, Zhe-Han Mo, Jia-Jun Wang3d1 minpaper
CADWorld: Computer-Use Benchmark for Long-Horizon Computer-Aided Design arxiv.org
CADWorld is a new benchmark suite of 200 long‑horizon mechanical CAD tasks in FreeCAD, covering sketching, part modeling, assembly, CAM, FEM, and more. Agents interact via screenshots and GUI actions; success is checked by executable validation of the saved CAD artifacts. Seven existing agents achieve at most 17.5 % success versus an 87 % expert baseline, highlighting the gap between GUI competen…
Hugging Face Daily Papersarxiv.orgZihan Dong, Yuanzhe Liu, Zhiyuan Ma8d1 minpaper
VideoGen-Agent: Reinforcing Video Generation Agents arxiv.org
VideoGen-Agent is a multimodal RL‑trained agent that orchestrates augmentation, generation, and verification tools to improve text‑to‑video synthesis on a new 600‑prompt benchmark (VABench). It lifts a base generator’s score from 56.5 to 75.6 (‑19.1 pts) and to 86.1 when the generation tools are upgraded, with 84.3% human preference over the strongest baseline.
Hugging Face Daily Papersarxiv.orgBinxu Li, Haoyi Duan, Yuhui Zhang1d1 minpaper
In-Context Robot Learning with VLM Agents arxiv.org
GPT‑Policy is a framework that lets a large vision‑language model (e.g. GPT‑6 Astra) perform in‑context robot learning: a context compiler extracts visual transitions from demos, the VLM proposes tool actions, and a constrained controller verifies and executes them. Real‑robot experiments show that raw video demos improve success rates even without explicit action labels, and that providing align…
Hugging Face Daily Papersarxiv.orgDongzhou Cheng, Taoran Yi, Ye Fang6d1 minpaper
HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harness arxiv.org
HarnessVLN introduces a zero‑shot, training‑free embodied navigation framework that wraps a multimodal LLM in an "Agent Harness" – a tool‑based protocol that validates planner actions against spatial evidence, tracks progress with hierarchical event memory, and maintains a persistent spatiotemporal graph for recovery. The system works for instruction‑following and object‑goal tasks, achieving 60.…
Hugging Face Daily Papersarxiv.orgYang Chen, Lirong Che, Zhenyu Huang8d1 minpaper
HuRo: Robotizing Human Videos for Scalable VLA Pretraining arxiv.org
The paper introduces a pipeline that converts heterogeneous human videos into robot‑aligned observations and actions, creating the 630K‑episode HuRo dataset. Pretraining vision‑language‑action (VLA) policies on this data boosts real‑world manipulation success from ~51% to ~80% and improves out‑of‑distribution robustness.
Hugging Face Daily Papersarxiv.orgJinho Jeong, Se June Joo, Jaehyun Kang4d1 minpaper

VABench: Measuring Embodied Spatial Intelligence through Visual Demonstrations, Active Perception, and Metric Control

Summary

Related reading

Transferring the Intelligence of VLMs to Robotic Controlarxiv.org

CADWorld: Computer-Use Benchmark for Long-Horizon Computer-Aided Designarxiv.org

VideoGen-Agent: Reinforcing Video Generation Agentsarxiv.org

In-Context Robot Learning with VLM Agentsarxiv.org

HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harnessarxiv.org

HuRo: Robotizing Human Videos for Scalable VLA Pretrainingarxiv.org

Transferring the Intelligence of VLMs to Robotic Control arxiv.org

CADWorld: Computer-Use Benchmark for Long-Horizon Computer-Aided Design arxiv.org

VideoGen-Agent: Reinforcing Video Generation Agents arxiv.org

In-Context Robot Learning with VLM Agents arxiv.org

HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harness arxiv.org

HuRo: Robotizing Human Videos for Scalable VLA Pretraining arxiv.org