proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersZihan Dong, Yuanzhe Liu, Zhiyuan Ma1 min readpaperintermediate

CADWorld: Computer-Use Benchmark for Long-Horizon Computer-Aided Design

Read on arxiv.org

Summary

CADWorld is a new benchmark suite of 200 long‑horizon mechanical CAD tasks in FreeCAD, covering sketching, part modeling, assembly, CAM, FEM, and more. Agents interact via screenshots and GUI actions; success is checked by executable validation of the saved CAD artifacts. Seven existing agents achieve at most 17.5 % success versus an 87 % expert baseline, highlighting the gap between GUI competen…

  • 200 tasks across 11 mechanical‑CAD workflow categories provide a comprehensive, structured evaluation of computer‑use agents.
  • Success criteria are executable checks on persistent FreeCAD artifacts, measuring geometry, constraints, parametric structure, manufacturing state, and simulation results.
  • Current agents perform poorly (≤17.5 % success), with failures shifting from artifact creation to structural and geometric correctness as capability improves.
  • The benchmark exposes a clear research gap: building agents that can reliably execute long‑horizon, verification‑driven engineering workflows.

Benchmarking realistic engineering workflows pushes AI agents beyond simple GUI navigation toward producing verifiable, domain‑specific artifacts, a prerequisite for practical automation in design and manufacturing.

7/10
machine learningbenchmarkengineeringcadfreecad

Related reading

  2. VABench: Measuring Embodied Spatial Intelligence through Visual Demonstrations, Active Perception, and Metric Control

    VA‑Bench is a new benchmark that evaluates general‑purpose multimodal LLMs on the full observe‑reason‑act‑revise loop in embodied robotics, using RGB demonstrations, active camera control, and metric Cartesian commands. The best model reaches 53.9% average task success, showing active perception helps but long‑horizon tasks remain unsolved.

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents

    The paper presents RecreationWorld, a five‑platform framework that lets hybrid computer‑use agents learn by recreating the behavior of a running reference, and introduces RecreationBench, a 250‑task benchmark with programmatic and visual assertions. Experiments show GPT‑6 Astra reaches 58.1% overall but struggles with deeper programmatic tests, highlighting gaps in current agents.

    Hugging Face Daily Papersarxiv.org2 minpaper

  4. SNAP3D: Physically Grounded 3D Parts for Assembly from a Single Image

    The paper presents SNAP3D, a physics‑guided pipeline that converts a single image into a set of 3D parts that can be assembled without interpenetration. By using simulation‑driven connector placement and a new physics‑based evaluation, the method yields assemblies that are both geometrically accurate and stable enough for 3D printing.

    Hugging Face Daily Papersarxiv.org1 minpaper

  5. The DevFest Community Workshop Experience: Building Real Agents Together

    Google’s DevFest Community Workshop introduced a “Workbench” format that emphasizes architectural mental models over copy‑paste code, guiding engineers to build long‑running, self‑evolving multi‑agent systems with the Agent Development Kit and Gemini Enterprise platforms. Attendees learned state‑separation, workflow pausing, and self‑patching pipelines, and the series will continue in five more c…

    Google Cloud Bloggoogle.com2 min

  6. Android Bench 2.0: Pushing the frontier with challenging long-horizon tasks

    Android Bench 2.0 adds a set of long‑horizon tasks (multi‑day Android development problems) and introduces agent‑based evaluation. Scoring is now continuous, with the best model achieving a 28 % pass rate on these tasks, far lower than the ~91 % on earlier short tasks. The post lists new models on the leaderboard and points to updated methodology and GitHub repo.

    Androidgoogleblog.com4 minHN2