proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersZhaochong An, Fei Zhang, Menglin Jia1 min readpaperadvanced

Native Action-Prior Learning from Videos for World Action Models

Read on arxiv.org

Summary

Native Action-Prior Learning (NAVA‑WAM) trains robot action policies directly from observation‑only videos by matching future video flow through a joint attention mechanism, then fine‑tunes with a small set of labeled demos. Experiments show it beats prior methods on both in‑distribution and out‑of‑distribution tasks and transfers to real robots with fewer action labels.

  • NAVA‑WAM pretrains action policies directly from observation‑only videos using future‑video flow‑matching and joint attention.
  • Training proceeds in two stages: video‑only pretraining then fine‑tuning with a small set of action‑labeled demos via joint video‑action flow matching.
  • Asymmetric attention decouples visual processing from iterative action denoising, enabling efficient action‑only inference.
  • Experiments report consistent outperformance over prior methods on in‑distribution and out‑of‑distribution tasks with fewer action labels.

Robotics engineers who want to reduce reliance on costly action‑annotated trajectories should care.

6/10
computer visionroboticscontrolvideo pretrainingaction prior

Related reading

  1. EVO-WAM: Evolving World Action Models through Video-Action Verification

    EVO-WAM adapts world action models to unseen robot tasks by generating video-action rollouts and self‑verifying them with vision-language and inverse dynamics models, removing the need for external execution or new demonstrations. It raises simulated RoboTwin 2.0 success rates from ~27% to 68% (Cosmos3) and ~28% to 46% (DreamZero), and boosts real‑world composite task success from 20% to 76.7%.

    Hugging Face Daily Papersarxiv.org1 minpaper

  2. HuRo: Robotizing Human Videos for Scalable VLA Pretraining

    The paper introduces a pipeline that converts heterogeneous human videos into robot‑aligned observations and actions, creating the 630K‑episode HuRo dataset. Pretraining vision‑language‑action (VLA) policies on this data boosts real‑world manipulation success from ~51% to ~80% and improves out‑of‑distribution robustness.

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models

    Latent Interface Training (LIT) first learns a goal‑conditioned action prior without visual input, then adds a pose‑supervised latent interface as the only visual conditioning path. Applied to several vision‑language‑action models, LIT cuts vision‑action shortcuts and lifts LIBERO‑Plus success by 3.9–10.7 points and real‑world task success by 13.3–16.7 points under distribution shifts.

    Hugging Face Daily Papersarxiv.org1 minpaper

  4. Dream4ACT: A Shared Visual Action Interface for Multi-Embodiment Video-Action Modeling

    Dream4ACT introduces a shared visual action interface (“action views”) that renders robot joint configurations from four virtual cameras using URDF forward kinematics, letting a single video autoencoder and diffusion transformer model both observations and actions across different robot embodiments. Trained with masked flow‑matching, the model attains 88.98% success on RoboTwin 2.0 and a 65.66 ov…

    Hugging Face Daily Papersarxiv.org1 minpaper

  5. Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation

    Ego2Act introduces a new benchmark for evaluating goal-directed manipulation in egocentric video generation, featuring 2,640 videos across 110 real-world tasks. It reveals that current video generation models struggle with multi-step physical reasoning, often skipping steps and failing at fine-grained object manipulation and persistent world modeling.

    Hugging Face Daily Papersarxiv.org1 minpaper

  6. In-Context Robot Learning with VLM Agents

    GPT‑Policy is a framework that lets a large vision‑language model (e.g. GPT‑6 Astra) perform in‑context robot learning: a context compiler extracts visual transitions from demos, the VLM proposes tool actions, and a constrained controller verifies and executes them. Real‑robot experiments show that raw video demos improve success rates even without explicit action labels, and that providing align…

    Hugging Face Daily Papersarxiv.org1 minpaper