proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersGehao Zhang, Weikai Huang, Shailesh Shailesh1 min readpaperadvanced

Grounded Action Model: 3D Grounding as a Foundation for Robotics

Read on arxiv.org

Summary

The Grounded Action Model (GAM) adds explicit 3D metric grounding to robot foundation models via a shared object‑centric representation, improving robustness to scene changes. Experiments on RoboTwin 2.0, LIBERO‑PRO, and real robots show state‑of‑the‑art success rates, especially under visual shift and long‑horizon tasks.

  • GAM converts language, point, or box prompts into a unified object‑centric embedding that captures geometry and visual features.
  • A multi‑stream transformer fuses this embedding with robot state history to predict short action chunks.
  • On RoboTwin 2.0, GAM reaches 55.3% success (vs 52% Spatial Forcing) and 47.6% under scene randomization (vs 30.4% Abot‑M0).
  • On LIBERO‑PRO, GAM attains 61% average success across 16 perturbations, outperforming the previous best (53%).

Robotics engineers building manipulation systems should care because explicit 3D grounding dramatically improves generalization and enables reliable low‑level control for high‑level planners.

7/10
roboticsfoundation modelsmanipulationtransformers3d visionrobot learning

Related reading

  1. ActionPiece: Rethinking Action Tokenization for Autoregressive Vision-Language-Action Models

    Action tokenizers for autoregressive VLA models often fail to preserve subtle action adjustments, despite good pointwise reconstruction. This paper introduces Physical Rank Consistency (PRC) to measure relational fidelity and ActionPiece, a new tokenizer that uses joint supervision to preserve these physical relationships, significantly improving policy success on robotics benchmarks.

    Hugging Face Daily Papersarxiv.org1 minpaper

  2. Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models

    Latent Interface Training (LIT) first learns a goal‑conditioned action prior without visual input, then adds a pose‑supervised latent interface as the only visual conditioning path. Applied to several vision‑language‑action models, LIT cuts vision‑action shortcuts and lifts LIBERO‑Plus success by 3.9–10.7 points and real‑world task success by 13.3–16.7 points under distribution shifts.

    Hugging Face Daily Papersarxiv.org1 minpaper

  4. MintAct: A Unified Visual Agent for Digital Environments

    MintAct is a family of vision-language models (2B-8B) that unifies UI grounding, multi-step navigation, and visual tool use across mobile, desktop, and web environments. It achieves state-of-the-art performance by leveraging a scalable environment and an asynchronous reinforcement learning infrastructure.

    Hugging Face Daily Papersarxiv.org1 minpaper

  5. HuRo: Robotizing Human Videos for Scalable VLA Pretraining

    The paper introduces a pipeline that converts heterogeneous human videos into robot‑aligned observations and actions, creating the 630K‑episode HuRo dataset. Pretraining vision‑language‑action (VLA) policies on this data boosts real‑world manipulation success from ~51% to ~80% and improves out‑of‑distribution robustness.

    Hugging Face Daily Papersarxiv.org1 minpaper

  6. Zing-0.5: Toward Playable Worlds with Real-Time Joint Action and Text Control

    Zing‑0.5 is a 5 B autoregressive world model that lets users control generated environments in real time using both keyboard actions and text prompts. The paper introduces unified action‑text conditioning, segment‑level teacher distillation, and a low‑cost streaming inference pipeline that runs at 24 FPS (832×480) for about $0.009 per minute, achieving 81 % overall and 88.5 % consistency on a nav…

    Hugging Face Daily Papersarxiv.org1 minpaper