proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersJeonghye Kim, Minseon Kim, Young Jin Kim1 min readpaperintermediate

ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks

Read on arxiv.org

Summary

ProgramDistill is a new benchmark that automatically extracts 1,975 replay‑verified feature behaviors from 26 real web apps, builds 4,063 coding‑agent tasks, and measures how well state‑of‑the‑art agents (e.g., GPT‑6 Astra, Claude Opus 5) can reconstruct full or partial applications, revealing steep drops in success as restoration depth grows.

  • The mine‑craft‑patch pipeline factorizes a working web app into granular, replayable features without any human labeling.
  • ProgramDistill yields 4 k+ tasks across 26 apps, providing a scalable, difficulty‑controlled benchmark for coding agents.
  • Evaluation of nine frontier agents shows GPT‑6 Astra reaches only 49 % success on full‑app reconstruction, while Claude Opus 5 lags at 29 %.
  • Partial‑app reconstruction success degrades sharply with depth: from 100 % to 64 % (GPT‑6) and 96 % to 32 % (Claude) when restoring 1→8 features.

Current coding‑agent evaluations rely on static prompts or issue descriptions, which miss the reality of inferring behavior from existing software. ProgramDistill bridges that gap by forcing agents to learn from concrete, executable reference behavior, exposing their limitations in incremental code…

7/10
aisoftware engineeringbenchmarkweb developmentcoding agents

Related reading

  1. RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents

    The paper presents RecreationWorld, a five‑platform framework that lets hybrid computer‑use agents learn by recreating the behavior of a running reference, and introduces RecreationBench, a 250‑task benchmark with programmatic and visual assertions. Experiments show GPT‑6 Astra reaches 58.1% overall but struggles with deeper programmatic tests, highlighting gaps in current agents.

    Hugging Face Daily Papersarxiv.org2 minpaper

  2. Your Agent Aced the Task. Will It Do It Again?

    The post introduces the Consistency Analyzer, a cheap black‑box diagnostic that flags flip‑prone decision steps in LLM agent traces, and shows how feeding the resulting consistency guidelines back into ALTK‑Evolve halves the gap between mean success and all‑run success (Pass⁵) on the AppWorld benchmark without hurting average accuracy.

    Hugging Facehuggingface.co8 minHN21

  3. Cloudflare Introduces the Agent Development Lifecycle to Replace Traditional SDLC

    Cloudflare’s Agent Development Lifecycle (ADLC) replaces the classic SDLC with an autonomous, event‑driven workflow system that lets AI agents handle code generation, testing, deployment, and maintenance. The platform builds on Cloudflare Workflows and the new @cloudflare/ci library to spin up containers, run headless browsers, and chain steps with caching and credential support. Observability is…

    InfoQinfoq.com2 min

  4. Building a Reliable Script-to-Video Pipeline for Modern Web Applications

    A practical guide to building a reliable, observable script‑to‑video pipeline. It advocates a staged architecture (scene manifest → asset generation → voice → rendering → encoding), validates AI‑generated JSON with Zod, persists project and scene state, uses explicit workflow states, runs heavy work in background queues, limits concurrency, and stores voice, captions, and assets separately to ena…

    SitePointsitepoint.com11 min

  5. Grounded Skill Synthesis from Code at Scale for Agentic Intelligence

    Code2Skill is an automated pipeline that mines popular GitHub repositories to extract verifiable, implementation‑anchored procedural “skills”. It builds a bank of ~1 M skill records (atomic ops, workflows, patterns) with provenance metadata, verifies each via blind reconstruction, and shows that augmenting LLM‑based agents with these skills yields an average 11.7% performance lift across 72 proto…

    Hugging Face Daily Papersarxiv.org1 minpaper

  6. CodeMidas: Scaling Agentic Coding RL Environments from Code Itself

    CodeMidas builds RL environments directly from open‑source code: agents explore a repo, infer a spec, generate tests from the original implementation, and filter tasks via execution checks. The pipeline yields 5,545 high‑quality coding tasks across 23 languages and 15 domains. Training the MiMo‑V2.5 agent with GRPO on this dataset improves benchmark scores by 8‑18% (e.g., DeepSWE +11.7%, ProgramB…

    Hugging Face Daily Papersarxiv.org1 minpaper