proomt

Search

Search posts, papers, and topics

Hall of Fame

Hall of FameJohn Graham-Cumming201922 min readpostmortemintermediate

Details of the Cloudflare outage on July 2, 2019

Read on cloudflare.com

Summary

A Cloudflare WAF Managed Rule introduced a regex with catastrophic backtracking, exhausting CPU on every core and causing a 27‑minute global outage. The team used synthetic tests, CPU metrics, and a global termination mechanism to halt the rule, then rolled it back after confirming the fix.

  • A single poorly‑written regular expression can cause massive CPU backtracking and saturate a globally distributed service.
  • Global termination can instantly disable a component, but access‑control failures can delay its execution.
  • Internal authentication outages (Access down) and reliance on rarely‑used bypass paths hindered rapid remediation.
  • Real‑time synthetic health checks and per‑core CPU monitoring were essential for early detection of the failure.

Operators of large, globally distributed systems need to understand the hidden performance risks of complex regexes and ensure robust, accessible rollback mechanisms.

7/10
observabilityincident responseregexpostmortemwaf

Related reading

  2. Cloudflare Introduces the Agent Development Lifecycle to Replace Traditional SDLC

    Cloudflare’s Agent Development Lifecycle (ADLC) replaces the classic SDLC with an autonomous, event‑driven workflow system that lets AI agents handle code generation, testing, deployment, and maintenance. The platform builds on Cloudflare Workflows and the new @cloudflare/ci library to spin up containers, run headless browsers, and chain steps with caching and credential support. Observability is…

    InfoQinfoq.com2 min

  3. When scanners miss the attack: how Cloudflare Client-Side Security protects storefronts

    Cloudflare’s Page Shield uses a graph‑neural‑network (GNN) to model JavaScript as a syntax‑tree graph, followed by a lightweight LLM for second‑opinion triage and an ensemble of frontier models for deep analysis. This pipeline caught eight malicious payloads across four distinct affiliate‑theft and backdoor techniques that traditional scanners missed, demonstrating the need for runtime, behavior‑…

    Cloudflarecloudflare.com21 minHN2

  5. Cloudflare/Security-Audit-Skill

    Cloudflare open‑sources a “security‑audit” skill that turns an LLM‑enabled coding agent into a structured vulnerability auditor. It runs six deterministic phases, validates findings against a JSON schema, and supports additive runs to improve coverage.

    Hacker News front pagegithub.com3 minreleaseHN20938
  6. 1 points

    Saving another 100TB of RAM with math (and Rust)

    Cloudflare reduced the memory footprint of its Pingora Backend Router by re‑examining the consistent‑hashing implementation in the pingora‑ketama library. By increasing the number of virtual hash points per server from the default 1 to the standard 160 (and applying weighted hashing based on disk capacity), they cut the per‑node overhead enough to reclaim >100 TB of RAM across the fleet. The post…

    Hacker News front pagecloudflare.com13 minHN478120lobste.rs33