Hugging Face Daily PapersRui Liu, Bhavin Jawade, Haoqi LiSep 30, 20261 min readpaperadvanced

Align Then Reason: A Multimodal Lip-Sync Judge for Dubbing

Summary

Align Then Reason (ATR) is a multilingual lip-sync judge for dubbing that works without dubbed audio. It first aligns lip movements to phonetic units, then uses an LLM to reason about content and timing, achieving significant AUC improvements over baselines across seven languages and downstream tasks.

Dubbing quality control requires a reference-free judge using only silent video and text, as dubbed audio may not exist.
Existing visual speech recognizers and video-language models are insensitive to temporal errors in lip-sync.
ATR employs a two-stage approach: monotonic alignment of frame-level lip representations to phonetic units, followed by LLM reasoning.
An alignment scorer provides the LLM with both local and global evidence for joint content and timing judgment.

This method is crucial for automated, reference-free quality control in multilingual dubbing workflows, enabling efficient and accurate lip-sync evaluation before audio is even generated.

8/10

llm machine learning multimodal computer vision nlp speech processing

Related reading

WISE-ATTA: When to Ask for Labels in Budgeted Active Test-Time Adaptation arxiv.org
WISE-ATTA addresses budgeted active test-time adaptation (ATTA), where labels are only available for a fraction of test batches. It proposes a budget-aware strategy to decide when to request supervision over time, prioritizing useful periods and using drift-based sample selection.
Hugging Face Daily Papersarxiv.orgMuhammad Huzaifa, Lea Schönherr, Thorsten Eisenhofer6d1 minpaper
LEAP: Learned Block-wise Evidence Retrieval for Long Audio-Video Perception arxiv.org
LEAP is a framework for long audio-video question answering that addresses context limits by retrieving relevant evidence blocks. It uses a two-stage process of block-wise localization and a bounded answer pass, achieving significant performance improvements on AVQA benchmarks.
Hugging Face Daily Papersarxiv.orgJuyi Lin, Zhiqiang Lao, Jiali Cui5d1 minpaper
Almost Human, Except When It Matters: VoxParity and the Decisions a Voice Should Change arxiv.org
The paper introduces VoxParity, a benchmark that swaps audio cues while keeping transcripts fixed to evaluate whether voice agents act on what they hear. Across 183 scenarios, most current systems fail to adjust their actions, exposing a strong bias toward text‑only reasoning.
Hugging Face Daily Papersarxiv.orgBhavik Mangla7d1 minpaper
onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction arxiv.org
onPanda is an interactive annotation tool that lets humans correct LLM outputs token‑by‑token, then resumes generation from the corrected prefix. In a controlled study it cut median annotation time by 52% and the authors release a token‑level correction dataset (Panda‑CVL) for on‑policy fine‑tuning.
Hugging Face Daily Papersarxiv.orgLei Yang, Mengyin Liu, Jia Wang14d1 minpaper
OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning arxiv.org
OmniSeek is a framework that turns a large language model into a multi‑turn audio‑visual reasoning agent by letting it decide when to look or listen and which temporal windows to fetch as evidence. The authors train it on a synthetic 170K trajectory dataset, then refine with reinforcement learning and an Audio‑Visual Necessity loss, reporting consistent gains on several benchmarks.
Hugging Face Daily Papersarxiv.orgHaibo Wang, Jiteng Mu, Jialu Li4d1 minpaper
Do Audio LLMs Listen Before They Act? Diagnosing Acoustic-Context Gating in Voice Agents arxiv.org
The authors present VGBench, a 1,018‑item diagnostic suite that tests whether audio LLMs correctly mute actions when the acoustic context shifts (e.g., speaker switches). Training with VoxGate dramatically improves gating, raising mute rates from <15% to >90% while preserving tool‑call accuracy.
Hugging Face Daily Papersarxiv.orgYanjie Zhang, Nanchen Hu, Yushi Sun9d1 minpaper

Align Then Reason: A Multimodal Lip-Sync Judge for Dubbing

Summary

Related reading

WISE-ATTA: When to Ask for Labels in Budgeted Active Test-Time Adaptationarxiv.org

LEAP: Learned Block-wise Evidence Retrieval for Long Audio-Video Perceptionarxiv.org

Almost Human, Except When It Matters: VoxParity and the Decisions a Voice Should Changearxiv.org

onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correctionarxiv.org

OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoningarxiv.org

Do Audio LLMs Listen Before They Act? Diagnosing Acoustic-Context Gating in Voice Agentsarxiv.org

WISE-ATTA: When to Ask for Labels in Budgeted Active Test-Time Adaptation arxiv.org

LEAP: Learned Block-wise Evidence Retrieval for Long Audio-Video Perception arxiv.org

Almost Human, Except When It Matters: VoxParity and the Decisions a Voice Should Change arxiv.org

onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction arxiv.org

OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning arxiv.org

Do Audio LLMs Listen Before They Act? Diagnosing Acoustic-Context Gating in Voice Agents arxiv.org