proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersBhavana Akkiraju, Ravi Sastry Kolluru, Sri Charan D1 min readpaperadvanced

VākQA: A Benchmark and Evaluation Study for Telugu Spoken Factoid Question Answering

Read on arxiv.org

Summary

The authors present VākQA, a 2,001‑question spoken factoid QA benchmark for Telugu with audio, transcriptions, and human‑verified answers, and they validate automatic evaluation methods against human ratings. Using this setup they show that translation loses cultural nuance, ASR errors alter meaning, and cascaded ASR‑MT errors degrade model performance.

  • VākQA offers 2,001 Telugu factoid QA pairs (2.53 h audio) with bilingual transcriptions and human‑verified reference answers.
  • Human studies find Gemini-as-a-judge best matches human scores but is stricter; open‑weight judges penalize correct answers with different surface forms.
  • Experiments reveal translation erases cultural specificity, speech input introduces phonetic confusions, and ASR‑MT error cascades compound performance loss.
  • Both proprietary and open‑weight models underperform humans, especially on spoken input and domain‑specific phrasing.

Anyone building multilingual spoken QA systems needs a realistic Telugu benchmark and reliable evaluation methods to gauge real‑world performance.

7/10
nlpbenchmarkspeech processingquestion answeringmultilingual

Related reading

  1. Building and Evaluating Fixed-Voice Thai TTS from Synthetic Speech

    Researchers train a compact 82 M‑parameter Thai fixed‑voice TTS model using synthetic speech generated by a large voice‑cloning teacher, requiring only a 15‑second real reference. The student achieves 68.2% keyword accuracy and 91.4% pause precision, outperforming its teacher on pause placement and enabling on‑device inference.

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue

    OmniVChat defines native audio‑visual dialogue where a model consumes raw audio and video streams and replies in text. The authors build OmniVChat‑Studio, a multi‑agent simulator that generates single‑ and multi‑turn audio‑visual conversations, and use it to create OmniVChat‑Bench, a benchmark covering five dialogue abilities. They also propose OmniVChat‑RL, a reinforcement‑learning reward that b…

    Hugging Face Daily Papersarxiv.org2 minpaper

  5. Article: Your Next DSL Author Is a Language Model

    Typed Domain Grounding (TDG) embeds a DSL inside a mainstream language the LLM already knows (e.g., Kotlin) and uses the host compiler as an oracle. The author describes five building blocks—embedding, choosing a host language with high training‑data frequency, compiler‑driven type safety, a generate‑compile‑repair loop, and an on‑demand teaching tool—and shows measured results from kUML, a Kotli…

    InfoQinfoq.com18 min

  6. Geometry of Values: Task Vector Composition for Ethical Preference Alignment in Language Models

    The authors release a 12k‑instance multilingual dilemma dataset (English + Hindi, Arabic, Spanish, Chinese) covering three pairwise value conflicts (Honesty‑Justice, Justice‑Autonomy, Autonomy‑Honesty). Benchmarking GPT‑5‑mini shows a consistent Honesty‑over‑Autonomy bias across languages. Llama‑3.2‑1/3B models exhibit a first‑option bias that can be eliminated (>98% accuracy) via plain fine‑tuni…

    Hugging Face Daily Papersarxiv.org1 minpaper