Hall of FameDan Luu201510 min readintermediate
Reading postmortems
Summary
Dan Luu surveys public postmortems and finds that most severe outages stem from a handful of recurring causes—poor error‑handling code, risky configuration changes, hardware faults, manual processes, and missing monitoring/alerting. He backs the claims with study numbers and argues engineers should focus on testing, automation, and observability to cut these failure modes.
- Bad error‑handling code causes cascading failures; 92% of critical failures in a study were due to mishandled errors.
- Configuration changes account for roughly half of public postmortems and often lack proper testing before rollout.
- Hardware faults, including silent DRAM errors, occur far more frequently than advertised specifications.
- Manual operational steps are a common source of outages; automating repeatable procedures can eliminate many human‑error cases.
Ops and reliability engineers should prioritize robust error handling, safe config deployment, hardware awareness, automation, and observability to reduce outage risk.
6/10
