Hugging Face Daily PapersBin Lin, Bo Zhao, Boyang Wang2 min readpaperadvanced
StepAudio 3 Gen Technical Report
Summary
StepAudio 3 Gen is a general‑purpose audio generation model that replaces diffusion with a discrete autoregressive generator over residual vector quantization tokens. Using a 16‑layer RVQ tokenizer and progressive pretraining, it reaches state‑of‑the‑art zero‑shot TTS and voice‑design performance while handling speech, vocals, sound effects, and music.
- Uses a discrete autoregressive generator over RVQ tokens instead of diffusion for audio synthesis.
- Tokenizer encodes audio at 12.5 Hz into a shared 16 × 2048 residual code space, preserving semantic and waveform information.
- First codebook is predicted autoregressively over time; a lightweight causal Transformer fills the remaining 15 codebooks across the codebook axis.
- Interference‑aware progressive pretraining lets the model acquire audio capabilities while retaining the LLM's textual abilities.
Engineers building multi‑modal generative systems need a unified, token‑based approach that scales across audio domains and integrates with large language models.
8/10