proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersYiran Wang, Xingyilang Yin, Junfu Pu1 min readpaperadvanced

GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay

Read on arxiv.org

Summary

The paper presents GameHorizon, a unified suite comprising an automated annotation pipeline, a 5,000‑hour multi‑horizon gameplay dataset from 21 AAA titles, and reproducible offline and online benchmarks. Using it, the authors evaluate 47 models, exposing a hierarchy of task difficulty and gaps in long‑term planning.

  • GameHorizon‑Annotator automates creation of multi‑horizon language instructions aligned with video and action streams.
  • GameHorizon‑Data offers 5,000 h of AAA gameplay from 21 titles with temporally aligned video, actions, and instructions.
  • GameHorizon‑Bench supplies reproducible offline QA and stepwise online evaluation that links scores to real gameplay performance.
  • Benchmarking 47 models reveals a clear hierarchy of task difficulty and gaps in long‑horizon planning.

Game AI and multimodal model researchers should care because it provides the first large‑scale, multi‑horizon benchmark for reliably comparing gameplay capabilities across models.

7/10
machine learningcomputer visiongamingbenchmarkdatasets

Related reading

  1. AI for Games in the Foundation Model Era

    The paper surveys how foundation models are used across six roles in the game development lifecycle—from playing agents to design assistance and runtime adaptation. It highlights limited transferability due to game-specific interfaces and notes that evaluation is mature for bounded play but weak for adaptive and testing scenarios.

    Hugging Face Daily Papersarxiv.org1 minpaper

  2. Android Bench 2.0: Pushing the frontier with challenging long-horizon tasks

    Android Bench 2.0 adds a set of long‑horizon tasks (multi‑day Android development problems) and introduces agent‑based evaluation. Scoring is now continuous, with the best model achieving a 28 % pass rate on these tasks, far lower than the ~91 % on earlier short tasks. The post lists new models on the leaderboard and points to updated methodology and GitHub repo.

    Androidgoogleblog.com4 minHN2

  3. RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents

    The paper presents RecreationWorld, a five‑platform framework that lets hybrid computer‑use agents learn by recreating the behavior of a running reference, and introduces RecreationBench, a 250‑task benchmark with programmatic and visual assertions. Experiments show GPT‑6 Astra reaches 58.1% overall but struggles with deeper programmatic tests, highlighting gaps in current agents.

    Hugging Face Daily Papersarxiv.org2 minpaper

  4. Emergence World: Adversarial Stress-Testing of Long-Horizon Multi-Agent Systems

    Emergence World is a continuously running multi‑agent sandbox used to stress‑test frontier LLM‑based agents over weeks. Eight parallel worlds (seven homogeneous, one mixed) generated 850 k LLM calls and ~50 B tokens while agents pursued goals, used tools, and maintained persistent memory. The authors injected three adversarial events—prompt injection, misinformation, and private‑memory exposure—a…

    Hugging Face Daily Papersarxiv.org1 minpaper

  5. CADWorld: Computer-Use Benchmark for Long-Horizon Computer-Aided Design

    CADWorld is a new benchmark suite of 200 long‑horizon mechanical CAD tasks in FreeCAD, covering sketching, part modeling, assembly, CAM, FEM, and more. Agents interact via screenshots and GUI actions; success is checked by executable validation of the saved CAD artifacts. Seven existing agents achieve at most 17.5 % success versus an 87 % expert baseline, highlighting the gap between GUI competen…

    Hugging Face Daily Papersarxiv.org1 minpaper

  6. ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks

    ProgramDistill is a new benchmark that automatically extracts 1,975 replay‑verified feature behaviors from 26 real web apps, builds 4,063 coding‑agent tasks, and measures how well state‑of‑the‑art agents (e.g., GPT‑6 Astra, Claude Opus 5) can reconstruct full or partial applications, revealing steep drops in success as restoration depth grows.

    Hugging Face Daily Papersarxiv.org1 minpaper