Hugging FaceEvelyn Duesterwald, Lilian Ngweta, Vatche Isahagian, Jayaram Radhakrishnan, Vinod Muthusamy, Gaodan Fang, Ashwath Vaithinathan Aravindan, Punleuk Oum, G Thomas, Merve Unuvar8 min readintermediate
Your Agent Aced the Task. Will It Do It Again?
Summary
The post introduces the Consistency Analyzer, a cheap black‑box diagnostic that flags flip‑prone decision steps in LLM agent traces, and shows how feeding the resulting consistency guidelines back into ALTK‑Evolve halves the gap between mean success and all‑run success (Pass⁵) on the AppWorld benchmark without hurting average accuracy.
- Mean@k hides a large *consistency gap*: a ReAct GPT‑4.1 agent scores 77.4 % Mean@5 but only 53.0 % Pass⁵ (24.4 pp gap).
- Consistency Analyzer resamples each decision point of a single recorded trajectory (k=5 completions by default) to compute a per‑step flip probability, requiring only one extra model call per step and no ground‑truth la…
- Guidelines generated from the analyzer’s high‑risk steps are injected at inference time via ALTK‑Evolve, raising Pass⁵ from 53 % to 69 % (gap ↓ from 24.4 pp to 12.0 pp) while Mean@5 stays ~81 %.
- Improvements are strongest on medium and hard tasks (+22.9 pp and +14.3 pp Pass⁵ respectively) and generalize to similar tasks and a weaker 120B model.
Reliability matters for real‑world agents (e.g., financial reconciliation) where a user expects the same answer on repeated identical queries. Pass⁵ surfaces this risk, and the Consistency Analyzer provides a practical, low‑cost way to turn instability into reusable, model‑agnostic guidelines, clos…
7/10
