proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersBin Lin, Bo Zhao, Boyang Zhang2 min readpaperintermediate

StepAudio 3 Realtime Technical Report

Read on arxiv.org

Summary

StepAudio 3 Realtime is an audio‑language foundation model that runs a continuous listen‑converse‑think‑act loop. It introduces Deep Perception for rich acoustic cue extraction, Seamless Duplex for handling pauses/back‑channels, and a Think‑While‑Speaking mechanism that lets the model reason in parallel with speech output. On benchmarks it scores 73.0 macro avg on StepAudioChat, 90.6 on MMSU, 98.…

  • Listen‑converse‑think‑act loop enables continuous, low‑latency interaction.
  • Think‑While‑Speaking decouples reasoning from speech generation, preserving real‑time responsiveness.
  • Seamless Duplex models synchronized audio streams to naturally handle interruptions and back‑channels.
  • Performance on multiple audio‑dialogue benchmarks approaches or exceeds dedicated reasoning models.

Real‑time spoken AI systems have to balance deep reasoning with sub‑second latency. StepAudio 3 demonstrates a concrete architecture that achieves both, which is directly relevant for building voice assistants, interactive agents, and any system where users expect fluid turn‑taking.

6/10
llmaudiorealtimedialoguespeech

Related reading

  1. StepAudio 3 Gen Technical Report

    StepAudio 3 Gen is a general‑purpose audio generation model that replaces diffusion with a discrete autoregressive generator over residual vector quantization tokens. Using a 16‑layer RVQ tokenizer and progressive pretraining, it reaches state‑of‑the‑art zero‑shot TTS and voice‑design performance while handling speech, vocals, sound effects, and music.

    Hugging Face Daily Papersarxiv.org2 minpaperHN2

  3. Zing-0.5: Toward Playable Worlds with Real-Time Joint Action and Text Control

    Zing‑0.5 is a 5 B autoregressive world model that lets users control generated environments in real time using both keyboard actions and text prompts. The paper introduces unified action‑text conditioning, segment‑level teacher distillation, and a low‑cost streaming inference pipeline that runs at 24 FPS (832×480) for about $0.009 per minute, achieving 81 % overall and 88.5 % consistency on a nav…

    Hugging Face Daily Papersarxiv.org1 minpaper

  4. OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue

    OmniVChat defines native audio‑visual dialogue where a model consumes raw audio and video streams and replies in text. The authors build OmniVChat‑Studio, a multi‑agent simulator that generates single‑ and multi‑turn audio‑visual conversations, and use it to create OmniVChat‑Bench, a benchmark covering five dialogue abilities. They also propose OmniVChat‑RL, a reinforcement‑learning reward that b…

    Hugging Face Daily Papersarxiv.org2 minpaper

  5. Building and Evaluating Fixed-Voice Thai TTS from Synthetic Speech

    Researchers train a compact 82 M‑parameter Thai fixed‑voice TTS model using synthetic speech generated by a large voice‑cloning teacher, requiring only a 15‑second real reference. The student achieves 68.2% keyword accuracy and 91.4% pause precision, outperforming its teacher on pause placement and enabling on‑device inference.

    Hugging Face Daily Papersarxiv.org1 minpaper

  6. FRAUDSkill: Structured Frozen-Weight Skill Optimization for Audio Anti-Fraud Detection

    FRAUDSkill is a framework that keeps a pretrained audio‑language model frozen and learns an external layer of skill programs, routing policies, and decision rules to meet a structured anti‑fraud detection protocol. On the TeleAntiFraud benchmark it reaches 73.5% Macro‑F1 (≈32% improvement) while cutting invalid predictions to 1.94%.

    Hugging Face Daily Papersarxiv.org1 minpaper