proomt

Search

Search posts, papers, and topics

All posts

TwilioMaria Bermudez11 min readintermediate

How we AI-updated 1,138 Error-code Docs Pages

Read on twilio.com

Summary

Twilio built a two‑phase pipeline that first scores each error‑code doc with a deterministic script, then feeds only low‑scoring pages to an LLM (gpt‑5.4) using a strict prompt. The model’s diff is applied via git, linted, and opened as a PR for a reviewer. In 12 weeks they updated 1,138 pages (≈34.6 s/page) with a human reviewing every change, cutting manual effort from an estimated 18 k hours a…

  • Score docs with a simple, deterministic script (structure 50, content 35, resources 15) and only send pages < 70 to the LLM to avoid unnecessary edits.
  • Use a prompt that (a) enforces a direct instructional voice, (b) forbids fabrications, (c) caps added links, and (d) allows the model to reply "I can't find anything to add" when no improvement is needed.
  • Apply model output as a git diff (single hunk) and run a lint step before opening a PR; keep batch size small (30 → 100 pages) so reviewers can stay on top of changes.
  • Maintain a tracking file to avoid duplicate PRs and to skip files with local modifications.

Accurate error‑code documentation directly reduces support tickets and improves developer experience. Twilio’s systematic, AI‑assisted approach shows how to scale doc maintenance for large, fast‑moving API catalogs while preserving quality through human review.

6/10
devopsllmautomationaitestingdocumentation

Related reading

  1. Five models pass the easy code reviews. Then the diffs get hard.

    A hands‑on benchmark of five LLM code‑review models (Claude Opus 5, GPT 5‑6 Sol, Grok 4.6, Kimi 3, GLM 5.3, Qwen 3.8 Max, DeepSeek v4 Pro) shows that all models ace tiny 15‑line diffs but diverge sharply on 100‑300‑line, cross‑file changes. The author built a deterministic harness (single HTTP call, fixed system prompt, JSON output) and measured detection rate, false‑positives, latency, token usa…

    Bjørn Johansenbjornjohansen.com16 min

  2. Changing the game: How Google uses agentic AI to secure hundreds of millions of lines of code

    Google’s AI & Infrastructure team built an agentic pipeline (Mantis) that runs pre‑submit AI‑driven scans on every code check‑in, validates findings with a fast triage agent (AST + call‑graph analysis) achieving >92% precision in <1 min, then auto‑generates fixes via a bug‑fix agent. Localized threat models and a two‑step scan cut false‑positives to ~3% and prevent hundreds of vulnerabilities eac…

    Google Cloud Bloggoogle.com4 min

  3. Eliminating AI Code Hallucinations with TypeScript Compiler Diagnostics and TDD Loops

    The article shows how to replace raw TypeScript compiler output with a programmatic JSON feedback loop that merges diagnostics and Vitest test failures, feeding the structured data back to an LLM coding agent for self‑correction. It provides concrete code for extracting, enriching, validating, and de‑duplicating errors, and demonstrates token savings and deterministic iteration until compilation…

    SitePointsitepoint.com15 min

  4. We now auto-approve and merge 15% of PRs

    This article describes how a team implemented a bot to auto-approve and merge 15% of low-risk pull requests, aiming to reduce code review bottlenecks and improve development velocity. The system relies on custom deterministic linters, basic end-to-end tests, and a risk scoring mechanism to identify safe changes.

    Swizec Tellerswizec.com3 min

  6. How Meshtastic Metrics Exporter Turned Eight Prometheus Queries Into One with Tiger Data

    A Python exporter reads Meshtastic mesh telemetry from MQTT and writes it directly into a TimescaleDB‑enabled PostgreSQL instance, replacing a Prometheus setup that hit ~0.5 M series for 10 k nodes. The single database lets Grafana dashboards join node metadata with eight time‑series tables in one query, eliminates scrape‑budget limits, and uses native retention/compression (14‑day compression, 3…

    Timescaletigerdata.com6 min