Android4 min readintermediate
Android Bench 2.0: Pushing the frontier with challenging long-horizon tasks
Summary
Android Bench 2.0 adds a set of long‑horizon tasks (multi‑day Android development problems) and introduces agent‑based evaluation. Scoring is now continuous, with the best model achieving a 28 % pass rate on these tasks, far lower than the ~91 % on earlier short tasks. The post lists new models on the leaderboard and points to updated methodology and GitHub repo.
- Long‑horizon tasks (LHTs) simulate real‑world Android work that can take days, covering dependency upgrades, new feature builds, full app creation, and cross‑platform porting.
- Scoring moved from binary pass/fail to a continuous completion rate that blends functional correctness, visual fidelity, and regression avoidance.
- Top LHT pass rate is ~28 %, compared with ~91 % on the original benchmark, highlighting the difficulty of multi‑step, architectural work for current LLMs.
- Agentic evaluation is added: models are run through their native agents (e.g., Gemini 3.8 Flash via Google Antigravity) to measure end‑to‑end developer workflow impact.
The benchmark surfaces concrete weaknesses of current LLMs in large‑scale Android refactoring and migration tasks, informing both model developers and engineering teams about realistic expectations when automating multi‑day development work.
6/10
