proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersXu Xu, Jinxiu Liu, Zhangbo Qiao1 min readpaperadvanced

OmniHarness: Harnessing Generalizable Visual Generation via Symbolic Policy Learning

Read on arxiv.org

Summary

OmniHarness introduces a symbolic‑policy framework that extracts reusable procedural knowledge from multimodal LLM‑driven visual generation runs. By decoupling task logic from instance inputs, the system can instantiate, adapt, and compose policies for new visual tasks, using intermediate verification for on‑the‑fly refinement while keeping the underlying model frozen. Self‑directed practice task…

  • Symbolic policies capture shared procedures across visual generation tasks, improving generalization beyond task‑specific distillation.
  • Intermediate verification enables dynamic failure recovery and policy refinement without updating model weights.
  • Self‑directed practice expands the policy library before downstream tasks appear, leading to measurable performance gains.
  • Frozen policy snapshots can be dropped into existing visual agents, yielding immediate improvements.

Current multimodal LLM pipelines embed task‑specific heuristics in model weights, limiting transfer to new visual generation problems and requiring costly fine‑tuning. OmniHarness shows that abstracting execution traces into reusable symbolic policies can achieve strong zero‑shot generalization whi…

8/10
llmmachine learningmultimodalml infravisual generationsymbolic policy

Related reading

  1. OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue

    OmniVChat defines native audio‑visual dialogue where a model consumes raw audio and video streams and replies in text. The authors build OmniVChat‑Studio, a multi‑agent simulator that generates single‑ and multi‑turn audio‑visual conversations, and use it to create OmniVChat‑Bench, a benchmark covering five dialogue abilities. They also propose OmniVChat‑RL, a reinforcement‑learning reward that b…

    Hugging Face Daily Papersarxiv.org2 minpaper

  2. Designer-RSI: Evolving Procedural Memory from User Traffic for Agentic Graphic Design

    The paper introduces Designer‑RSI, a continual‑adaptation system that couples a frozen design‑software‑controlling LLM with an external procedural memory of natural‑language design skills. Over five adaptation rounds on real user briefs, the memory grows from 76 to 139 procedures and lifts execution success from 72.7% to 99.3%, showing that skill accumulation and selective replay can dramatically…

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. In-Context Robot Learning with VLM Agents

    GPT‑Policy is a framework that lets a large vision‑language model (e.g. GPT‑6 Astra) perform in‑context robot learning: a context compiler extracts visual transitions from demos, the VLM proposes tool actions, and a constrained controller verifies and executes them. Real‑robot experiments show that raw video demos improve success rates even without explicit action labels, and that providing align…

    Hugging Face Daily Papersarxiv.org1 minpaper

  4. OmniVBench: A Benchmark and Large-Scale Dataset for Omni Reference-to-Video Generation

    OmniVBench is a new benchmark and the Omni‑R2V Dataset, offering 7 task families, 18 fine‑grained reference‑to‑video generation tasks and a factor‑grounded evaluation checklist of over 12 k items. The dataset provides 340 k industrial‑grade video samples and pipelines for constructing reference‑target pairs, exposing large performance gaps in current R2V models.

    Hugging Face Daily Papersarxiv.org2 minpaper

  5. HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harness

    HarnessVLN introduces a zero‑shot, training‑free embodied navigation framework that wraps a multimodal LLM in an "Agent Harness" – a tool‑based protocol that validates planner actions against spatial evidence, tracks progress with hierarchical event memory, and maintains a persistent spatiotemporal graph for recovery. The system works for instruction‑following and object‑goal tasks, achieving 60.…

    Hugging Face Daily Papersarxiv.org1 minpaper

  6. Can MiniMax-H3 Reason About the Physical World? An Evaluation of Omni-Modal Generative Model

    This paper introduces a novel evaluation framework to assess the physical world reasoning capabilities of omni-modal generative models like MiniMax-H3. It found that MiniMax-H3 achieved an overall success rate of 41.97% across 517 instances, with significant performance variations depending on the input modalities and reasoning tasks.

    Hugging Face Daily Papersarxiv.org2 minpaper