proomt

Search

Search posts, papers, and topics

Hall of Fame

Hall of FameJim Gray198527 min readpaperadvanced

Why Do Computers Stop and What Can Be Done About It?

Read on azurewebsites.netHN 23·6 comments

Summary

Jim Gray analyzes failure reports from Tandem NonStop systems, showing that administration and software bugs cause most outages while hardware is a minor factor. He argues that modular redundancy, process‑pairs, and transaction mechanisms give software the same high availability as hardware redundancy.

  • In a fault‑tolerant deployment, 40%+ of outages stem from admin actions and software bugs, not hardware failures.
  • Modularity + fail‑fast modules + spare replicas turn a module failure into an instantaneous repair, dramatically lowering MTTR.
  • Process‑pair execution combined with transactions provides a practical software fault‑tolerance primitive.
  • Most software faults are soft (transient); detecting and restarting fail‑fast modules recovers quickly.

Engineers building high‑availability services should apply these software‑centric fault‑tolerance patterns rather than relying solely on hardware redundancy.

8/10
architecturedistributed systemstransactionsreliabilityfault tolerance

Related reading

  1. Reading postmortems

    Dan Luu surveys public postmortems and finds that most severe outages stem from a handful of recurring causes—poor error‑handling code, risky configuration changes, hardware faults, manual processes, and missing monitoring/alerting. He backs the claims with study numbers and argues engineers should focus on testing, automation, and observability to cut these failure modes.

    Hall of Famedanluu.com10 minHN2

  2. Metastable Failures in Distributed Systems

    This paper introduces and formalizes "metastable failures" in distributed systems, a class of outages where a trigger pushes a system into a bad state that persists due to a sustaining effect, even after the trigger is removed. These failures often stem from features designed for efficiency or reliability and require significant external intervention to resolve.

    Hall of Famesigops.org25 minpaperHN16112

  3. Simple Testing Can Prevent Most Critical Failures

    A study of 198 production failures in distributed systems (Cassandra, HDFS, etc.) found that 92% of catastrophic outages stemmed from incorrect handling of non-fatal errors. Over half of these could have been prevented by simple testing of error handling code, even without deep system understanding.

    Hall of Fameusenix.org58 minpaperHN4

  4. The Night Watch

    This satirical essay argues that systems programmers are the most crucial engineers, especially for post-apocalyptic survival, due to their experience debugging low-level hardware and distributed systems. It humorously contrasts the brutal realities of systems debugging with the perceived softer challenges of fields like HCI and machine learning.

    Hall of Fameusenix.org12 minHN20533

  5. The Tail at Scale

    The paper defines “tail‑tolerant” systems that keep interactive latency (≤100 ms) low even as services scale to thousands of servers, and surveys the main causes of latency outliers. It shows that techniques such as request replication, hedged requests, and resource isolation can dramatically shrink the latency tail with modest overhead, enabling higher utilization without over‑provisioning.

    Hall of Fameresearch.google1 minpaperHN2

  6. How Complex Systems Fail

    Cook argues that complex system accidents arise from multiple interacting faults, making the notion of a single root cause meaningless, and that hindsight bias skews post‑incident analysis. Understanding failure as a social construct shifts focus to systemic improvement.

    Hall of Famecomplexsystems.fail2 minHN38979