Bjørn Johansen16 min readintermediate
Five models pass the easy code reviews. Then the diffs get hard.
Summary
A hands‑on benchmark of five LLM code‑review models (Claude Opus 5, GPT 5‑6 Sol, Grok 4.6, Kimi 3, GLM 5.3, Qwen 3.8 Max, DeepSeek v4 Pro) shows that all models ace tiny 15‑line diffs but diverge sharply on 100‑300‑line, cross‑file changes. The author built a deterministic harness (single HTTP call, fixed system prompt, JSON output) and measured detection rate, false‑positives, latency, token usa…
- Tiny diffs (≤30 lines) are not a useful benchmark for LLM code reviewers; all competent models pass.
- Real‑world code reviews need multi‑file, 100‑+ line diffs to expose differences in reasoning depth, token limits, and timeout behavior.
- OpenRouter model IDs may route to multiple hosts; pinning the provider (e.g., `provider: { only: [host] }`) is essential for reproducible measurements.
- Token caps matter: models that exceed the cap (DeepSeek, Qwen) truncate answers, reducing detection rates.
LLM‑powered code review is being adopted in CI pipelines, but engineers lack hard data on how model size, routing, and token limits affect reliability. This benchmark quantifies detection rates, false‑positive noise, latency, and cost across a realistic set of diffs, exposing hidden failure modes (…
8/10
