Apple Machine Learning ResearchSep 17, 20261 min readpaperadvanced

REVERSAL-BENCH: A Reversibility Axis and Reset Oracle for Measuring the Reset-Free RL Cliff

Summary

The paper presents REVERSAL‑BENCH, a benchmark that varies environment reversibility with a parameter ρ and provides a ground‑truth reset oracle for eight manipulation tasks. Using it, the authors show that reset‑free RL agents hit a sharp reversibility cliff and become permanently trapped, while episodic agents remain robust.

REVERSAL‑BENCH defines a continuous reversibility parameter ρ and a reset oracle to label recoverable vs irrecoverable states across eight manipulation tasks.
Experiments reveal a sharp “reversibility cliff”: reset‑free agents become permanently trapped as ρ increases, while episodic agents continue learning.
The cliff is caused by irreversibility itself, confirmed by comparing to geometrically reversible task variants.
A safety shield can predict irreversible states but only prevents failure when it can steer the agent away; it cannot recover once trapped.

Researchers building autonomous RL systems for real‑world manipulation should care because irreversibility can halt learning without external resets.

8/10

reinforcement learning robotics benchmark simulation safety

Related reading

Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agents arxiv.org
EvoSkill‑GUI lets GUI agents revise their procedural skills on‑the‑fly without extra training by using a reflect‑revise‑reuse loop that edits skill packages during execution. The approach yields up to +16.2% improvement on MobileWorld and similar gains on AndroidWorld and OSWorld, and the evolved skills transfer to related tasks.
Hugging Face Daily Papersarxiv.orgBofan Chen, Boxuan Zhang, Fei Tang7d1 minpaper
From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention arxiv.org
The paper presents PARTS, a framework that augments a frozen pretrained robot policy with RL‑learned residuals on selected bottleneck subtasks, using local success rewards and minimal human resets. In real‑world bimanual and single‑arm tasks, PARTS more than doubles success rates with only minutes of robot rollouts, outperforming prior fine‑tuning methods.
Hugging Face Daily Papersarxiv.orgSichang Su, Benjamin Yang, Zhiyun Deng4d1 minpaper
RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents arxiv.org
The paper presents RecreationWorld, a five‑platform framework that lets hybrid computer‑use agents learn by recreating the behavior of a running reference, and introduces RecreationBench, a 250‑task benchmark with programmatic and visual assertions. Experiments show GPT‑6 Astra reaches 58.1% overall but struggles with deeper programmatic tests, highlighting gaps in current agents.
Hugging Face Daily Papersarxiv.orgShuai Bai, Jiayong Deng, Yikun Fu4d2 minpaper
GAVEL: Graph World Models for Verified and Efficient Long-Horizon LLM Task Planning arxiv.org
GAVEL augments LLM‑driven robot planners with an explicit graph world model that verifies actions, repairs violations, and reasons over belief distributions, boosting single‑task success from 41 % to 92 % and multi‑task success from 20 % to 93 % on BEHAVIOR‑1K.
Hugging Face Daily Papersarxiv.orgRuiyang Wang, Hao-Lun Hsu, Swarajh Mehta6d1 minpaper
DataFlex-RL: An Evaluation Platform for RLVR Data Policies arxiv.org
The paper introduces DataFlex‑RL, a platform to benchmark how different data‑selection policies affect reinforcement‑learning‑with‑verifiable‑rewards training. Across extensive experiments on Qwen2.5‑7B and Llama‑3.1‑8B, uniform sampling is the only method that consistently improves performance, and no alternative policy yields a statistically significant gain.
Hugging Face Daily Papersarxiv.orgHao Liang, Mingrui Chen, Hengyi Feng17d1 minpaper
Your Agent Aced the Task. Will It Do It Again?huggingface.co
The post introduces the Consistency Analyzer, a cheap black‑box diagnostic that flags flip‑prone decision steps in LLM agent traces, and shows how feeding the resulting consistency guidelines back into ALTK‑Evolve halves the gap between mean success and all‑run success (Pass⁵) on the AppWorld benchmark without hurting average accuracy.
Hugging Facehuggingface.coEvelyn Duesterwald, Lilian Ngweta, Vatche Isahagian, Jayaram Radhakrishnan, Vinod Muthusamy, Gaodan Fang, Ashwath Vaithinathan Aravindan, Punleuk Oum, G Thomas, Merve Unuvar7d8 minHN21

REVERSAL-BENCH: A Reversibility Axis and Reset Oracle for Measuring the Reset-Free RL Cliff

Summary

Related reading

Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agentsarxiv.org

From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Interventionarxiv.org

RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agentsarxiv.org

GAVEL: Graph World Models for Verified and Efficient Long-Horizon LLM Task Planningarxiv.org

DataFlex-RL: An Evaluation Platform for RLVR Data Policiesarxiv.org

Your Agent Aced the Task. Will It Do It Again?huggingface.co

Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agents arxiv.org

From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention arxiv.org

RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents arxiv.org

GAVEL: Graph World Models for Verified and Efficient Long-Horizon LLM Task Planning arxiv.org

DataFlex-RL: An Evaluation Platform for RLVR Data Policies arxiv.org