proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersMingfei Gao, Rui Tian, Haiming Gang1 min readpaperadvanced

MintAct: A Unified Visual Agent for Digital Environments

Read on arxiv.org

Summary

MintAct is a family of vision-language models (2B-8B) that unifies UI grounding, multi-step navigation, and visual tool use across mobile, desktop, and web environments. It achieves state-of-the-art performance by leveraging a scalable environment and an asynchronous reinforcement learning infrastructure.

  • MintAct unifies UI grounding, multi-step navigation, and visual tool use across diverse digital environments.
  • The models are scaled at 2B, 4B, and 8B parameters, matching specialist performance.
  • A scalable environment hosts hundreds of concurrent instances for data collection and online RL.
  • An asynchronous RL framework manages cross-domain training distribution and maintains stability.

This paper is significant for researchers and engineers building general-purpose AI agents, as it demonstrates a unified and scalable approach to visual interaction across diverse digital environments.

8/10
llmreinforcement learningcomputer visionai agentsui automation

Related reading

  2. HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harness

    HarnessVLN introduces a zero‑shot, training‑free embodied navigation framework that wraps a multimodal LLM in an "Agent Harness" – a tool‑based protocol that validates planner actions against spatial evidence, tracks progress with hierarchical event memory, and maintains a persistent spatiotemporal graph for recovery. The system works for instruction‑following and object‑goal tasks, achieving 60.…

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. Zing-0.5: Toward Playable Worlds with Real-Time Joint Action and Text Control

    Zing‑0.5 is a 5 B autoregressive world model that lets users control generated environments in real time using both keyboard actions and text prompts. The paper introduces unified action‑text conditioning, segment‑level teacher distillation, and a low‑cost streaming inference pipeline that runs at 24 FPS (832×480) for about $0.009 per minute, achieving 81 % overall and 88.5 % consistency on a nav…

    Hugging Face Daily Papersarxiv.org1 minpaper

  4. Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL

    ActObs is a supervised fine-tuning method that trains agents to predict both actions and environmental observations from trajectories. This joint supervision improves subsequent reinforcement learning performance, leading to better exploration and task completion on benchmarks like Terminal-Bench 2.0 and aider-polyglot.

    Hugging Face Daily Papersarxiv.org1 minpaper

  5. Agent Anomaly Detection, now in Private Preview on the Gemini Enterprise Agent Platform

    Google’s Gemini Enterprise Agent Platform now offers a private‑preview Agent Anomaly Detection service that asynchronously analyzes agents’ OpenTelemetry traces and logs to flag risky behavior (e.g., tool misuse, privilege abuse) without adding latency, surfacing findings in Security Command Center and via an API for automated mitigation.

    Google Developersgoogleblog.com4 min

  6. In-Context Robot Learning with VLM Agents

    GPT‑Policy is a framework that lets a large vision‑language model (e.g. GPT‑6 Astra) perform in‑context robot learning: a context compiler extracts visual transitions from demos, the VLM proposes tool actions, and a constrained controller verifies and executes them. Real‑robot experiments show that raw video demos improve success rates even without explicit action labels, and that providing align…

    Hugging Face Daily Papersarxiv.org1 minpaper