Hall of FameDing Yuan et al.201458 min readpaperintermediate
Simple Testing Can Prevent Most Critical Failures
Summary
A study of 198 production failures in distributed systems (Cassandra, HDFS, etc.) found that 92% of catastrophic outages stemmed from incorrect handling of non-fatal errors. Over half of these could have been prevented by simple testing of error handling code, even without deep system understanding.
- 92% of catastrophic failures result from incorrect handling of non-fatal errors explicitly signaled in software.
- 58% of catastrophic failures could have been prevented by simple testing of error handling code.
- 35% of catastrophic failures are due to trivial error handling bugs (empty, over-general abort, or FIXME/TODO comments).
- 77% of failures are reproducible with unit tests, and almost all on 3 or fewer nodes.
Engineers building and maintaining distributed systems should care, as this paper provides concrete evidence that focusing on robust error handling and simple testing can significantly improve system reliability.
8/10