Hugging FaceTuhin KunduOct 3, 202613 min readintermediate

The Agent Said It Was Done. The Database Disagreed.

Summary

ThinkingBox benchmarks AI agents by checking the final database state after tool calls, revealing that many LLM‑driven agents succeed on a single attempt but fail to repeat the correct outcome. Across 507 tasks run 20 times, models differ widely in consistency and cost per successful attempt, with Kimi‑K3 being broad but inconsistent and Claude Opus models being more reliable.

Tool calls alone don’t guarantee correct outcomes; the resulting database state must be verified.
ThinkingBox runs each task 20 times and reports pass@1, pass@20, and observed 20/20 to measure reliability.
High pass@1 scores can hide low consistency—Kimi‑K3 solves 93.9% at least once but only 13.4% consistently.
Claude Opus 5.5 improves headline accuracy but adds no consistency over Opus 5.

Teams deploying LLM agents that modify real data need to know not just if they can succeed once, but if they do so reliably and affordably.

7/10

llm performance agents benchmark cost

Related reading

Your Agent Aced the Task. Will It Do It Again?huggingface.co
The post introduces the Consistency Analyzer, a cheap black‑box diagnostic that flags flip‑prone decision steps in LLM agent traces, and shows how feeding the resulting consistency guidelines back into ALTK‑Evolve halves the gap between mean success and all‑run success (Pass⁵) on the AppWorld benchmark without hurting average accuracy.
Hugging Facehuggingface.coEvelyn Duesterwald, Lilian Ngweta, Vatche Isahagian, Jayaram Radhakrishnan, Vinod Muthusamy, Gaodan Fang, Ashwath Vaithinathan Aravindan, Punleuk Oum, G Thomas, Merve Unuvar20d8 minHN21
41 Experiments in 6 Days: What the Human Was Actually For tigerdata.com
Timescale used an AI agent to run 41 experiments in six days, improving LLM conversational memory on the LoCoMo benchmark. The core insight is that humans are crucial for validating metrics and objectives, preventing the agent from optimizing against flawed measurements.
Timescaletigerdata.comMatvey Arye7d8 min
2026 in LLMs (so far)simonwillison.net
The post recaps 2026 LLM milestones, noting that Claude Opus 4.5 and GPT‑5.1 made coding agents reliable enough for daily use, sparking AI‑driven side projects and a surge of sandboxing and agent‑security discussions. The author reflects on "AI mania", personal experiments, and the cultural impact on engineers.
Simon Willisonsimonwillison.net8d21 minHN5
Constraint Decay: The Fragility of LLM Agents in Backend Code Generation arxiv.org
A systematic evaluation of LLM agents generating multi‑file backend code shows a sharp drop in correctness when structural constraints (framework conventions, ORM usage, API contracts) are added. Across 100 tasks in 8 Python web frameworks, assertion pass rates fall ~27 points, with data‑layer bugs (bad queries, ORM violations) driving most failures. Mid‑size models cope with minimal frameworks (…
arXiv cs.SE (Software Engineering)arxiv.orgFrancesco Dente, Dario Satriani, Paolo Papotti15d1 minpaperHN287197
What we learned running seven AI coding models on real production work hostinger.com
Hostinger ran seven AI coding models on 12 real PRs, measuring match to engineer solutions, cost, latency, and error patterns. The top five models performed similarly, so speed, price, and review rigor matter more than raw quality scores.
Hostingerhostinger.comSimon Lim10d8 min
VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks arxiv.org
VeriHarness turns an LLM generator into an agentic verifier by adding a workspace, evidence tools, and verification skills that resolve disagreement and challenge consensus. Across five long‑horizon benchmarks it improves selection scores by ~6 points over single rollouts and releases 26k rollouts for future work.
Hugging Face Daily Papersarxiv.orgCaiqi Zhang, Rujun Han, Zifeng Wang5d1 minpaper

The Agent Said It Was Done. The Database Disagreed.

Summary

Related reading

Your Agent Aced the Task. Will It Do It Again?huggingface.co

41 Experiments in 6 Days: What the Human Was Actually Fortigerdata.com

2026 in LLMs (so far)simonwillison.net

Constraint Decay: The Fragility of LLM Agents in Backend Code Generationarxiv.org

What we learned running seven AI coding models on real production workhostinger.com

VeriHarness: Scaling Agentic Verification for Long-Horizon Tasksarxiv.org

41 Experiments in 6 Days: What the Human Was Actually For tigerdata.com

Constraint Decay: The Fragility of LLM Agents in Backend Code Generation arxiv.org

What we learned running seven AI coding models on real production work hostinger.com

VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks arxiv.org