Hugging Face Daily PapersHaibo Wang, Jiteng Mu, Jialu LiOct 1, 20261 min readpaperadvanced

OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning

Summary

OmniSeek is a framework that turns a large language model into a multi‑turn audio‑visual reasoning agent by letting it decide when to look or listen and which temporal windows to fetch as evidence. The authors train it on a synthetic 170K trajectory dataset, then refine with reinforcement learning and an Audio‑Visual Necessity loss, reporting consistent gains on several benchmarks.

The agent dynamically selects when to query audio or visual data and the length of the temporal segment, appending raw evidence back into its context for subsequent reasoning.
Training uses a synthetic corpus (OmniTraj‑170K) of multi‑hop chain‑of‑thought trajectories with interleaved audio‑visual evidence, followed by supervised fine‑tuning.
A two‑stage RL stage with verifiable rewards further optimizes the evidence‑seeking policy.
An Audio‑Visual Necessity objective penalizes single‑modality shortcuts, encouraging true multimodal reasoning.

Teams building multimodal AI agents will find the tool‑integration protocol and training recipe useful for enabling efficient, evidence‑driven reasoning across audio and video.

6/10

llm reinforcement learning multimodal vision audio visual

Related reading

OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue arxiv.org
OmniVChat defines native audio‑visual dialogue where a model consumes raw audio and video streams and replies in text. The authors build OmniVChat‑Studio, a multi‑agent simulator that generates single‑ and multi‑turn audio‑visual conversations, and use it to create OmniVChat‑Bench, a benchmark covering five dialogue abilities. They also propose OmniVChat‑RL, a reinforcement‑learning reward that b…
Hugging Face Daily Papersarxiv.orgHaolin He, Yunfei Chu, Qi Chen18d2 minpaper
OmniHarness: Harnessing Generalizable Visual Generation via Symbolic Policy Learning arxiv.org
OmniHarness introduces a symbolic‑policy framework that extracts reusable procedural knowledge from multimodal LLM‑driven visual generation runs. By decoupling task logic from instance inputs, the system can instantiate, adapt, and compose policies for new visual tasks, using intermediate verification for on‑the‑fly refinement while keeping the underlying model frozen. Self‑directed practice task…
Hugging Face Daily Papersarxiv.orgXu Xu, Jinxiu Liu, Zhangbo Qiao23d1 minpaper
HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harness arxiv.org
HarnessVLN introduces a zero‑shot, training‑free embodied navigation framework that wraps a multimodal LLM in an "Agent Harness" – a tool‑based protocol that validates planner actions against spatial evidence, tracks progress with hierarchical event memory, and maintains a persistent spatiotemporal graph for recovery. The system works for instruction‑following and object‑goal tasks, achieving 60.…
Hugging Face Daily Papersarxiv.orgYang Chen, Lirong Che, Zhenyu Huang22d1 minpaper
Almost Human, Except When It Matters: VoxParity and the Decisions a Voice Should Change arxiv.org
The paper introduces VoxParity, a benchmark that swaps audio cues while keeping transcripts fixed to evaluate whether voice agents act on what they hear. Across 183 scenarios, most current systems fail to adjust their actions, exposing a strong bias toward text‑only reasoning.
Hugging Face Daily Papersarxiv.orgBhavik Mangla8d1 minpaper
Scaffolding Minds: Optimizing Latent Visual Target Representations for Multimodal Reasoning arxiv.org
The paper pinpoints two weaknesses in latent‑reasoning pipelines—suboptimal visual token encoders and deterministic RL sampling—and proposes Scaffolding Minds, which trains a dedicated scaffolding encoder and a stochastic RL sampler (learning both mean and variance). This yields up to +19 points improvement on spatial planning tasks and +5.6 points average gain on nine visual reasoning benchmarks.
Hugging Face Daily Papersarxiv.orgHaoqiang Kang, Yinpeng Chen, Luyang Liu7d1 minpaper
AV-GRPO: Modality-Anchored Decoupling Diffusion Reinforcement Learning for Joint Audio-Video Generation arxiv.org
The paper presents AV‑GRPO, a diffusion‑based reinforcement‑learning framework that treats audio and video generation as separate but coordinated tasks by anchoring rollouts to each modality and freezing the opposite tower during optimization. Evaluated on the new 5DAV dataset and benchmarks, it achieves higher fidelity, better text‑modality alignment, and tighter audio‑video sync than the previo…
Hugging Face Daily Papersarxiv.orgZhiyu Xu, Weilong Yan, Yufei Shi12d1 minpaper

OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning

Summary

Related reading

OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialoguearxiv.org

OmniHarness: Harnessing Generalizable Visual Generation via Symbolic Policy Learningarxiv.org

HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harnessarxiv.org

Almost Human, Except When It Matters: VoxParity and the Decisions a Voice Should Changearxiv.org

Scaffolding Minds: Optimizing Latent Visual Target Representations for Multimodal Reasoningarxiv.org

AV-GRPO: Modality-Anchored Decoupling Diffusion Reinforcement Learning for Joint Audio-Video Generationarxiv.org

OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue arxiv.org

OmniHarness: Harnessing Generalizable Visual Generation via Symbolic Policy Learning arxiv.org

HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harness arxiv.org

Almost Human, Except When It Matters: VoxParity and the Decisions a Voice Should Change arxiv.org

Scaffolding Minds: Optimizing Latent Visual Target Representations for Multimodal Reasoning arxiv.org

AV-GRPO: Modality-Anchored Decoupling Diffusion Reinforcement Learning for Joint Audio-Video Generation arxiv.org