proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersShijie Lian, Bin Yu, Zhaolong Shen1 min readpaperadvanced

ActionPiece: Rethinking Action Tokenization for Autoregressive Vision-Language-Action Models

Read on arxiv.org

Summary

Action tokenizers for autoregressive VLA models often fail to preserve subtle action adjustments, despite good pointwise reconstruction. This paper introduces Physical Rank Consistency (PRC) to measure relational fidelity and ActionPiece, a new tokenizer that uses joint supervision to preserve these physical relationships, significantly improving policy success on robotics benchmarks.

  • Existing action tokenizers for VLA models struggle to preserve subtle action adjustments, even with low MSE, impacting policy performance.
  • Physical Rank Consistency (PRC) is a novel metric to evaluate how well tokenization preserves local physical distance rankings after reconstruction.
  • ActionPiece improves action tokenization by jointly supervising representation learning and quantization with physical rank preservation objectives.
  • It uses near-far ordering in encoder and quantized feature distances, and applies the same ordering to codeword assignment distributions.

Researchers and engineers developing autoregressive vision-language-action models should care, as improved action tokenization directly leads to more robust and capable robotic policies.

8/10
llmmachine learningreinforcement learningcomputer visionrobotics

Related reading

  1. In-Context Robot Learning with VLM Agents

    GPT‑Policy is a framework that lets a large vision‑language model (e.g. GPT‑6 Astra) perform in‑context robot learning: a context compiler extracts visual transitions from demos, the VLM proposes tool actions, and a constrained controller verifies and executes them. Real‑robot experiments show that raw video demos improve success rates even without explicit action labels, and that providing align…

    Hugging Face Daily Papersarxiv.org1 minpaper

  2. From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention

    The paper presents PARTS, a framework that augments a frozen pretrained robot policy with RL‑learned residuals on selected bottleneck subtasks, using local success rewards and minimal human resets. In real‑world bimanual and single‑arm tasks, PARTS more than doubles success rates with only minutes of robot rollouts, outperforming prior fine‑tuning methods.

    Hugging Face Daily Papersarxiv.org1 minpaper

  4. Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models

    Latent Interface Training (LIT) first learns a goal‑conditioned action prior without visual input, then adds a pose‑supervised latent interface as the only visual conditioning path. Applied to several vision‑language‑action models, LIT cuts vision‑action shortcuts and lifts LIBERO‑Plus success by 3.9–10.7 points and real‑world task success by 13.3–16.7 points under distribution shifts.

    Hugging Face Daily Papersarxiv.org1 minpaper

  6. HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harness

    HarnessVLN introduces a zero‑shot, training‑free embodied navigation framework that wraps a multimodal LLM in an "Agent Harness" – a tool‑based protocol that validates planner actions against spatial evidence, tracks progress with hierarchical event memory, and maintains a persistent spatiotemporal graph for recovery. The system works for instruction‑following and object‑goal tasks, achieving 60.…

    Hugging Face Daily Papersarxiv.org1 minpaper