proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersYuheng Shi, Xiaohuan Pei, Minjing Dong1 min readpaperadvanced

Region-Level Policy Optimization for Fine-grained MLLM Perception

Read on arxiv.org

Summary

Vision‑RL2 trains a lightweight proposal network via region‑level reinforcement learning to select high‑resolution evidence for multimodal LLMs, allowing coarse‑resolution localization and fine‑resolution recognition. Across six fine‑grained vision benchmarks it reduces visual token count by ~4× while matching or surpassing full‑resolution accuracy.

  • Localization tolerates 3–4× more token compression than recognition, enabling coarse‑view ROI detection.
  • Vision‑RL2 uses a frozen MLLM to score region removals, training the proposal network without region annotations.
  • Sparse encoding of selected regions amplifies evidence and discards background tokens, cutting memory and compute.
  • Evaluated on six fine‑grained benchmarks and four MLLM backbones, achieving up to 4× fewer visual tokens with equal or higher accuracy.

MLLM engineers needing fine‑grained visual reasoning under token‑budget constraints should care.

8/10
reinforcement learningvision languagecomputer visiontoken efficiencymultimodal llm

Related reading

  1. In-Context Robot Learning with VLM Agents

    GPT‑Policy is a framework that lets a large vision‑language model (e.g. GPT‑6 Astra) perform in‑context robot learning: a context compiler extracts visual transitions from demos, the VLM proposes tool actions, and a constrained controller verifies and executes them. Real‑robot experiments show that raw video demos improve success rates even without explicit action labels, and that providing align…

    Hugging Face Daily Papersarxiv.org1 minpaper

  2. DataFlex-RL: An Evaluation Platform for RLVR Data Policies

    The paper introduces DataFlex‑RL, a platform to benchmark how different data‑selection policies affect reinforcement‑learning‑with‑verifiable‑rewards training. Across extensive experiments on Qwen2.5‑7B and Llama‑3.1‑8B, uniform sampling is the only method that consistently improves performance, and no alternative policy yields a statistically significant gain.

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. OmniHarness: Harnessing Generalizable Visual Generation via Symbolic Policy Learning

    OmniHarness introduces a symbolic‑policy framework that extracts reusable procedural knowledge from multimodal LLM‑driven visual generation runs. By decoupling task logic from instance inputs, the system can instantiate, adapt, and compose policies for new visual tasks, using intermediate verification for on‑the‑fly refinement while keeping the underlying model frozen. Self‑directed practice task…

    Hugging Face Daily Papersarxiv.org1 minpaper

  4. Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models

    Latent Interface Training (LIT) first learns a goal‑conditioned action prior without visual input, then adds a pose‑supervised latent interface as the only visual conditioning path. Applied to several vision‑language‑action models, LIT cuts vision‑action shortcuts and lifts LIBERO‑Plus success by 3.9–10.7 points and real‑world task success by 13.3–16.7 points under distribution shifts.

    Hugging Face Daily Papersarxiv.org1 minpaper