proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersAdam Hung, Bardienus P. Duisterhof, Deva Ramanan1 min readpaperadvanced

Modality-Autoregressive World-Action Models

Read on arxiv.org

Summary

ModAR is a world‑action model that autoregressively denoises multiple future modalities (depth, DINO features, point tracks) before predicting actions, letting each prediction condition on earlier outputs. It outperforms prior WAMs, achieving higher success rates with ~20× fewer training FLOPs and no pretraining.

  • Autoregressive denoising of depth, DINO features, and point tracks before action prediction improves WAM performance.
  • Predicting future RGB images adds little benefit compared to the other modalities.
  • ModAR reaches 75% success versus 72% for a pre‑trained baseline while using ~20× fewer FLOPs.
  • Sequential generation consistently outperforms existing WAM formulations across data scales.

Robotics researchers and engineers building learning‑based controllers should care because ModAR shows how to leverage multimodal predictions efficiently, reducing compute while boosting task success.

8/10
reinforcement learningmultimodalroboticsdeep learningperception

Related reading

  1. Zing-0.5: Toward Playable Worlds with Real-Time Joint Action and Text Control

    Zing‑0.5 is a 5 B autoregressive world model that lets users control generated environments in real time using both keyboard actions and text prompts. The paper introduces unified action‑text conditioning, segment‑level teacher distillation, and a low‑cost streaming inference pipeline that runs at 24 FPS (832×480) for about $0.009 per minute, achieving 81 % overall and 88.5 % consistency on a nav…

    Hugging Face Daily Papersarxiv.org1 minpaper

  2. Can MiniMax-H3 Reason About the Physical World? An Evaluation of Omni-Modal Generative Model

    This paper introduces a novel evaluation framework to assess the physical world reasoning capabilities of omni-modal generative models like MiniMax-H3. It found that MiniMax-H3 achieved an overall success rate of 41.97% across 517 instances, with significant performance variations depending on the input modalities and reasoning tasks.

    Hugging Face Daily Papersarxiv.org2 minpaper

  3. ActionPiece: Rethinking Action Tokenization for Autoregressive Vision-Language-Action Models

    Action tokenizers for autoregressive VLA models often fail to preserve subtle action adjustments, despite good pointwise reconstruction. This paper introduces Physical Rank Consistency (PRC) to measure relational fidelity and ActionPiece, a new tokenizer that uses joint supervision to preserve these physical relationships, significantly improving policy success on robotics benchmarks.

    Hugging Face Daily Papersarxiv.org1 minpaper

  6. Introducing System One Models and Jev

    TypeSafe AI announced its first “System One” model, Jev, a non‑text‑generating LLM that outputs type‑safe structured decisions with calibrated probabilities. It claims 40‑200× lower latency (70‑500 ms) and 100‑500× lower cost versus frontier LLMs, no hallucinations, and parallel sampling. The post includes a side‑by‑side demo, a custom “workflow” benchmark comparing Jev to GPT‑5.6/6 and other mod…

    Hacker News front pagetypesafe.ai9 minHN1824480lobste.rs26