Hugging Face Daily PapersZhen Wang, Changpeng Wang, Zhe LiuSep 28, 20262 min readpaperadvanced

PanoVLN: Towards Effective Panoramic Vision-and-Language Navigation

Summary

PanoVLN is a novel approach for Vision-and-Language Navigation that effectively utilizes panoramic observations. It achieves significant performance gains by modifying action prediction, training supervision, and visual representation, outperforming prior SOTA on R2R-CE and RxR-CE benchmarks.

Panoramic observations in VLN require specific architectural and training adjustments beyond simple image replacement for effective use.
PanoVLN introduces Confidence-Guided Execution (CGE) for dynamic, longer-horizon action planning from a single panoramic view.
Training data for panoramic VLN should include frequent branching points and clear instructions to improve route selection capabilities.
Combining semantic and geometric features from RGB panoramas is crucial for understanding spatial relationships in wide-field views.

This work is important for researchers and engineers developing autonomous navigation systems, as it significantly advances the state-of-the-art in vision-and-language navigation using panoramic inputs.

8/10

llm machine learning computer vision ai robotics navigation

Related reading

HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harness arxiv.org
HarnessVLN introduces a zero‑shot, training‑free embodied navigation framework that wraps a multimodal LLM in an "Agent Harness" – a tool‑based protocol that validates planner actions against spatial evidence, tracks progress with hierarchical event memory, and maintains a persistent spatiotemporal graph for recovery. The system works for instruction‑following and object‑goal tasks, achieving 60.…
Hugging Face Daily Papersarxiv.orgYang Chen, Lirong Che, Zhenyu Huang22d1 minpaper
OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning arxiv.org
OmniSeek is a framework that turns a large language model into a multi‑turn audio‑visual reasoning agent by letting it decide when to look or listen and which temporal windows to fetch as evidence. The authors train it on a synthetic 170K trajectory dataset, then refine with reinforcement learning and an Audio‑Visual Necessity loss, reporting consistent gains on several benchmarks.
Hugging Face Daily Papersarxiv.orgHaibo Wang, Jiteng Mu, Jialu Li5d1 minpaper
PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection arxiv.org
PANORAMA introduces a VLM that grounds each phrase of a caption to pixel masks by selecting from a phrase‑conditioned pool of mask proposals, trained jointly with caption generation. It sets a new benchmark on the PanoCaps dataset and outperforms prior methods on several pixel‑level grounding tasks.
Hugging Face Daily Papersarxiv.orgSara Pieri, Evangelos Kazakos, Shizhe Chen20d1 minpaper
FAMOS: Feed-Forward 3D Articulation Modeling from Sparse Observations arxiv.org
FAMOS is a feed-forward model that predicts 3D articulation, including movable-part segmentation and joint parameters, from sparse, unordered point clouds. It leverages multiple observations through a Multi-state Articulation Transformer and a novel observed articulation span objective, outperforming existing baselines.
Hugging Face Daily Papersarxiv.orgKevin Qu, Tao Sun, Massimiliano Viola19d1 minpaper
LLaDA-UI: Bringing Block-wise Diffusion to Vision-Language GUI Agents arxiv.org
LLaDA‑UI is a 16.7 B‑parameter Mixture‑of‑Experts vision‑language model that uses block‑wise diffusion to generate GUI actions in parallel. It achieves state‑of‑the‑art performance on multiple GUI grounding and navigation benchmarks, beating larger Qwen models while keeping decoding latency low.
Hugging Face Daily Papersarxiv.orgZhangxuan Gu, Haoxing Chen, Qi Qin27d1 minpaper
HuRo: Robotizing Human Videos for Scalable VLA Pretraining arxiv.org
The paper introduces a pipeline that converts heterogeneous human videos into robot‑aligned observations and actions, creating the 630K‑episode HuRo dataset. Pretraining vision‑language‑action (VLA) policies on this data boosts real‑world manipulation success from ~51% to ~80% and improves out‑of‑distribution robustness.
Hugging Face Daily Papersarxiv.orgJinho Jeong, Se June Joo, Jaehyun Kang18d1 minpaper

PanoVLN: Towards Effective Panoramic Vision-and-Language Navigation

Summary

Related reading

HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harnessarxiv.org

OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoningarxiv.org

PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selectionarxiv.org

FAMOS: Feed-Forward 3D Articulation Modeling from Sparse Observationsarxiv.org

LLaDA-UI: Bringing Block-wise Diffusion to Vision-Language GUI Agentsarxiv.org

HuRo: Robotizing Human Videos for Scalable VLA Pretrainingarxiv.org

HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harness arxiv.org

OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning arxiv.org

PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection arxiv.org

FAMOS: Feed-Forward 3D Articulation Modeling from Sparse Observations arxiv.org

LLaDA-UI: Bringing Block-wise Diffusion to Vision-Language GUI Agents arxiv.org

HuRo: Robotizing Human Videos for Scalable VLA Pretraining arxiv.org