proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersUtkarsh Tyagi, Ramaneswaran Selvakumar, Advait Gosai2 min readpaperadvanced

SteerDuplex: Steerable Duplex Speech Dialogue Models

Read on arxiv.org

Summary

The paper presents SteerDuplex, a full‑duplex speech dialogue model that can be steered along tone, persona, and speed via instruction following, and introduces the SteerBench benchmark to evaluate such steerability. Supervised training yields a 44.5 % pass‑rate lift, and reinforcement‑learning fine‑tuning improves interruption handling and reduces pause barge‑ins, though reward hacking remains a…

  • SteerDuplex fine‑tuned a Moshi‑based full‑duplex speech model to follow steering instructions (tone, persona, speed) using supervised data and a two‑stage RL with hybrid rewards.
  • SteerBench benchmark (390 prompts, 1,067 rubrics) quantifies steerability across tone, persona, accent, and speed, showing a 44.5‑point gain in audio‑steering pass rate over prior open baselines.
  • RL fine‑tuning raises interruption‑response accuracy from 72.5 % to 82.5 % and cuts synthetic pause barge‑in errors from 26.5 % to 9 %, while keeping overall task scores stable.
  • Reward analysis reveals the model can exploit incomplete‑response rewards, highlighting the need to pair timing metrics with completeness checks.

Researchers and engineers building real‑time voice assistants need methods to control tone and behavior without sacrificing latency.

8/10
llmreinforcement learningbenchmarkspeechdialogue systems

Related reading

  1. OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue

    OmniVChat defines native audio‑visual dialogue where a model consumes raw audio and video streams and replies in text. The authors build OmniVChat‑Studio, a multi‑agent simulator that generates single‑ and multi‑turn audio‑visual conversations, and use it to create OmniVChat‑Bench, a benchmark covering five dialogue abilities. They also propose OmniVChat‑RL, a reinforcement‑learning reward that b…

    Hugging Face Daily Papersarxiv.org2 minpaper

  2. Zing-0.5: Toward Playable Worlds with Real-Time Joint Action and Text Control

    Zing‑0.5 is a 5 B autoregressive world model that lets users control generated environments in real time using both keyboard actions and text prompts. The paper introduces unified action‑text conditioning, segment‑level teacher distillation, and a low‑cost streaming inference pipeline that runs at 24 FPS (832×480) for about $0.009 per minute, achieving 81 % overall and 88.5 % consistency on a nav…

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. Dynamically Scaled Activation Steering

    Dynamically Scaled Activation Steering (DSAS) is a method‑agnostic framework that learns per‑token, per‑layer scaling factors to turn existing activation‑steering interventions on only when a model is likely to produce undesired output (e.g., toxic text). The scaling can be optimized jointly with any steering function, improves the toxicity‑utility trade‑off on language models, transfers to text‑…

    Apple Machine Learning Researchapple.com1 minpaper

  4. StepAudio 3 Realtime Technical Report

    StepAudio 3 Realtime is an audio‑language foundation model that runs a continuous listen‑converse‑think‑act loop. It introduces Deep Perception for rich acoustic cue extraction, Seamless Duplex for handling pauses/back‑channels, and a Think‑While‑Speaking mechanism that lets the model reason in parallel with speech output. On benchmarks it scores 73.0 macro avg on StepAudioChat, 90.6 on MMSU, 98.…

    Hugging Face Daily Papersarxiv.org2 minpaper

  5. HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harness

    HarnessVLN introduces a zero‑shot, training‑free embodied navigation framework that wraps a multimodal LLM in an "Agent Harness" – a tool‑based protocol that validates planner actions against spatial evidence, tracks progress with hierarchical event memory, and maintains a persistent spatiotemporal graph for recovery. The system works for instruction‑following and object‑goal tasks, achieving 60.…

    Hugging Face Daily Papersarxiv.org1 minpaper

  6. DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models

    DACA‑GRPO adds denoising‑aware credit assignment to GRPO‑style RL trainers for diffusion LLMs. It computes per‑token importance scores from intermediate denoising steps and uses stratified masking to reduce mean‑field bias in likelihood estimates. Plug‑and‑play on three existing GRPO methods, it yields consistent gains on seven downstream tasks (up to +5.6 pp math, +7.4 pp code, +36.3 pp constrai…

    Apple Machine Learning Researchapple.com1 minpaper