proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersEmbedding Team1 min readpaperadvanced

Ovis-Embedding: Pushing the Frontiers of Universal Omni-Modal Embeddings

Read on arxiv.org

Summary

Ovis-Embedding is a unified omni‑modal embedding model that shares a single backbone for text, image, video, and audio, trained with contrastive loss, homogeneous‑source sampling, and embedding distillation. It achieves state‑of‑the‑art results on several multimodal retrieval benchmarks while offering low‑rank, dimension‑flexible embeddings.

  • Uses a single pretrained Qwen‑omni backbone for all modalities, avoiding separate towers.
  • Homogeneous‑source sampling creates task‑consistent batches with informative in‑batch negatives.
  • Combines focal loss with similarity‑based embedding distillation from expert models.
  • Low‑rank feature decomposition at inference yields compact, dimension‑adjustable embeddings with minimal loss.

Engineers building cross‑modal retrieval or unified embedding systems should watch for this approach to reduce model fragmentation and improve performance.

6/10
machine learningmultimodalcontrastive learningembeddings

Related reading

  1. FLAT: Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generation

    FLAT is a pre‑training framework that encodes images and text into a shared 1D token sequence with variable length via nested dropout, enabling the same embeddings for cross‑modal retrieval and generation. It attains state‑of‑the‑art scores on COCO and Flickr30K for captioning, retrieval, and text‑to‑image generation, and supports interpolation and zero‑shot composed retrieval.

    Hugging Face Daily Papersarxiv.org1 minpaper

  2. OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue

    OmniVChat defines native audio‑visual dialogue where a model consumes raw audio and video streams and replies in text. The authors build OmniVChat‑Studio, a multi‑agent simulator that generates single‑ and multi‑turn audio‑visual conversations, and use it to create OmniVChat‑Bench, a benchmark covering five dialogue abilities. They also propose OmniVChat‑RL, a reinforcement‑learning reward that b…

    Hugging Face Daily Papersarxiv.org2 minpaper

  3. OmniEdu: Open Foundation Models for Learning and Teaching

    OmniEdu is an open family of LLMs (4B‑27B) fine‑tuned on a curated, capability‑balanced educational corpus (≈70k examples, 16M tokens) covering subject competence, curriculum grounding, diagnostic reasoning, and pedagogical scaffolding. Across model scales it improves on K‑12 benchmarks (K12‑Bench EM 63.12 %/F1 76.69 %, MathFish 85.89 %, EDUMATH 86.95 %, MathTutorBench Scaffold 78.74 %) and achie…

    Hugging Face Daily Papersarxiv.org1 minpaper

  4. UFO: Chain-of-Evaluation for Omni-Condition Alignment in Multi-Modal Image Generation

    UFO introduces an Atomized Chain‑of‑Evaluation (AEU) framework that breaks omni‑condition alignment in multi‑modal image generation into a sequential set of fine‑grained checks, achieving a 15.25 % boost in correlation with human judgments. The authors also release UFO‑Bench, a benchmark for testing how well models satisfy combined textual and visual conditions.

    Hugging Face Daily Papersarxiv.org1 minpaper

  5. OmniVBench: A Benchmark and Large-Scale Dataset for Omni Reference-to-Video Generation

    OmniVBench is a new benchmark and the Omni‑R2V Dataset, offering 7 task families, 18 fine‑grained reference‑to‑video generation tasks and a factor‑grounded evaluation checklist of over 12 k items. The dataset provides 340 k industrial‑grade video samples and pipelines for constructing reference‑target pairs, exposing large performance gaps in current R2V models.

    Hugging Face Daily Papersarxiv.org2 minpaper

  6. WeVisDoc: From Coverage to Capability for Robust End-to-End Document Parsing

    WeVisDoc introduces a two‑stage data‑centric pipeline for end‑to‑end document parsing. Stage I expands coverage using heterogeneous data and structure‑preserving degradations. Stage II probes the Stage I model with a held‑out set, clusters residual errors, and directs targeted data creation and token‑budget reallocation. The 4‑billion‑parameter model reaches 95.38 Overall on OmniDocBench v1.6 and…

    Hugging Face Daily Papersarxiv.org1 minpaper