proomt

Search

Search posts, papers, and topics

Hall of Fame

Hall of FameNathan Bronson, Aleksey Charapko, Abutalib Aghayev, Timothy Zhu202125 min readpaperintermediate

Metastable Failures in Distributed Systems

Read on sigops.orgHN 161·12 comments

Summary

This paper introduces and formalizes "metastable failures" in distributed systems, a class of outages where a trigger pushes a system into a bad state that persists due to a sustaining effect, even after the trigger is removed. These failures often stem from features designed for efficiency or reliability and require significant external intervention to resolve.

  • Metastable failures are sustained by feedback loops (e.g., work amplification) and do not resolve when the initial trigger is gone.
  • Features like request retries and look-aside caching, intended for efficiency, can create vulnerability to metastable states.
  • Systems can operate in a "vulnerable" state for extended periods before a trigger causes a transition to a metastable failure.
  • Recovery from a metastable state typically requires a strong corrective push, such as dramatically reducing load or rebooting.

This paper provides a crucial framework for understanding and addressing a class of severe, hard-to-diagnose outages in large-scale distributed systems that were previously only understood anecdotally.

8/10
architectureperformancedistributed systemsreliabilitypostmortemfailure modes

Related reading

  1. Simple Testing Can Prevent Most Critical Failures

    A study of 198 production failures in distributed systems (Cassandra, HDFS, etc.) found that 92% of catastrophic outages stemmed from incorrect handling of non-fatal errors. Over half of these could have been prevented by simple testing of error handling code, even without deep system understanding.

    Hall of Fameusenix.org58 minpaperHN4

  2. Why Do Computers Stop and What Can Be Done About It?

    Jim Gray analyzes failure reports from Tandem NonStop systems, showing that administration and software bugs cause most outages while hardware is a minor factor. He argues that modular redundancy, process‑pairs, and transaction mechanisms give software the same high availability as hardware redundancy.

    Hall of Fameazurewebsites.net27 minpaperHN236

  3. Reading postmortems

    Dan Luu surveys public postmortems and finds that most severe outages stem from a handful of recurring causes—poor error‑handling code, risky configuration changes, hardware faults, manual processes, and missing monitoring/alerting. He backs the claims with study numbers and argues engineers should focus on testing, automation, and observability to cut these failure modes.

    Hall of Famedanluu.com10 minHN2

  4. Worker Backpressure (Part 1)

    Canva added a lightweight, local backpressure loop to its queue worker library that monitors per‑message success/failure, computes a backoff factor against a configurable failure‑rate set‑point, and throttles the worker’s concurrency. In two real incidents the mechanism kept failure rates under 2 % fleet‑wide, limited DLQ growth to a handful of messages, and maintained throughput without manual i…

    Canvacanva.dev10 min

  5. Best practices for handling cloud reliability incidents

    The article outlines a structured Verify→Investigate→Report→Resolve→Review workflow for GCP reliability incidents and stresses pre‑incident preparation across design, data, playbooks, and training. It lists concrete tools (Cloud Logging, Service Health, Gemini Assist) and reporting steps to help engineers reduce outage impact.

    Google Cloud Bloggoogle.com11 min

  6. How Uber Protects Against Retry Storms

    Uber developed a context-aware mechanism to prevent retry storms in deep microservice dependency chains. It introduces "error ownership" where services claim errors they originate and unclaim errors they propagate, allowing upstream callers to make informed retry decisions and avoid amplifying load on already struggling services.

    Hacker News front pageuber.com12 minHN11949