Hugging Face Daily PapersHaolin He, Yunfei Chu, Qi Chen2 min readpaperadvanced
OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue
Summary
OmniVChat defines native audio‑visual dialogue where a model consumes raw audio and video streams and replies in text. The authors build OmniVChat‑Studio, a multi‑agent simulator that generates single‑ and multi‑turn audio‑visual conversations, and use it to create OmniVChat‑Bench, a benchmark covering five dialogue abilities. They also propose OmniVChat‑RL, a reinforcement‑learning reward that b…
- Native audio‑visual input removes the need for separate speech‑to‑text or captioning pipelines, reducing latency and preserving perceptual cues.
- OmniVChat‑Studio synthesizes realistic multi‑modal dialogues using multiple agents, enabling large‑scale data generation where real recordings are scarce.
- OmniVChat‑Bench evaluates five core abilities (e.g., context understanding, visual grounding, multimodal reasoning) with metrics beyond keyword matching.
- OmniVChat‑RL reward combines task correctness, response latency, and stylistic quality, and can be applied via RL fine‑tuning of existing LLMs.
The work tackles two practical bottlenecks for multimodal assistants: lack of large, high‑quality audio‑visual dialogue data and reliable evaluation metrics. By providing a scalable data synthesis pipeline and a benchmark that captures nuanced multimodal reasoning, the paper offers a reproducible f…
7/10