CanvaMikalai Barysau10 min readintermediate
Worker Backpressure (Part 1)
Summary
Canva added a lightweight, local backpressure loop to its queue worker library that monitors per‑message success/failure, computes a backoff factor against a configurable failure‑rate set‑point, and throttles the worker’s concurrency. In two real incidents the mechanism kept failure rates under 2 % fleet‑wide, limited DLQ growth to a handful of messages, and maintained throughput without manual i…
- A per‑worker feedback loop (signals → controller → permits) can adaptively throttle workers without a central coordinator.
- Using a configurable failure‑rate set‑point (e.g., 5 %) lets the system stay at full speed under normal conditions and back off only when errors exceed background noise.
- The controller’s backoff factor is a single float (0.0‑1.0) that scales the number of concurrent polls, costing only two arithmetic ops per message.
- Real‑world data: during a 4‑hour cloud‑provider outage, fleet‑wide failure peaked at 1.42 % and DLQ grew by 1 message; during a 32.5‑hour sustained overload, fleet‑wide failure peaked at 3.7 % and only 22 messages hit t…
At scale, greedy workers amplify downstream failures, causing cascading overload, DLQ storms, and on‑call fatigue. A self‑regulating backpressure mechanism protects shared dependencies, reduces noise in incident response, and preserves throughput without manual scaling or complex coordination.
8/10
