Constraint Decay: The Fragility of LLM Agents in Backend Code Generation

Summary

A systematic evaluation of LLM agents generating multi‑file backend code shows a sharp drop in correctness when structural constraints (framework conventions, ORM usage, API contracts) are added. Across 100 tasks in 8 Python web frameworks, assertion pass rates fall ~27 points, with data‑layer bugs (bad queries, ORM violations) driving most failures. Mid‑size models cope with minimal frameworks (…

• Constraint decay: Adding architectural and data‑layer constraints reduces LLM‑generated code correctness by ~27 pp on average.
• Framework sensitivity: Performance varies widely; models do better on lightweight frameworks (Flask) than on convention‑heavy ones (FastAPI, Django).
• Error profile: Most failures stem from data‑layer defects—incorrect query composition and ORM runtime violations.
• Evaluation methodology: Dual assessment using end‑to‑end behavioral tests plus static verification across 80 greenfield and 20 feature‑addition tasks.

Production backends must obey strict architectural patterns, database schemas, and ORM contracts. If LLM agents cannot reliably respect these constraints, their utility for real‑world software engineering remains limited, despite functional correctness on simple prompts.