proomt

Search

Search posts, papers, and topics

Hall of Fame

Hall of FameAmazon Web Services20174 min readpostmortemadvanced

Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region

Read on amazon.comHN 1246·515 comments

Summary

An S3 team member accidentally removed too many servers in US-EAST-1 due to an incorrect command input, leading to a multi-hour outage of S3 and dependent AWS services. The incident highlighted challenges in restarting critical subsystems at scale and the need for improved operational safeguards.

  • An incorrect command input by an S3 team member removed critical capacity from the index and placement subsystems, causing the outage.
  • Restarting core S3 subsystems took longer than expected due to S3's massive growth and infrequent full restarts in large regions.
  • The outage impacted S3 GET/LIST/PUT/DELETE requests and dependent AWS services like EC2 launches, EBS, and Lambda.
  • Remediation includes modifying operational tools with safeguards, reprioritizing subsystem partitioning, and making the Service Health Dashboard multi-region.

This postmortem offers critical insights for engineers designing and operating large-scale distributed systems on the importance of robust operational tooling, rapid recovery mechanisms, and understanding system dependencies.

8/10
distributed systemsawsreliabilitys3postmortemoperations

Related reading

  1. AWS Cannot Restore Data Held Only in Damaged Middle East Availability Zones

    AWS admitted it cannot recover data that was stored only in the damaged Availability Zones of its Middle East (UAE) and Bahrain regions after Iranian drone strikes. The outage highlights that AWS’s multi‑AZ guarantees only protect against localized failures, not regional destruction, and that customers must implement cross‑region replication to meet durability and data‑residency requirements.

    InfoQinfoq.com5 min

  2. Reading postmortems

    Dan Luu surveys public postmortems and finds that most severe outages stem from a handful of recurring causes—poor error‑handling code, risky configuration changes, hardware faults, manual processes, and missing monitoring/alerting. He backs the claims with study numbers and argues engineers should focus on testing, automation, and observability to cut these failure modes.

    Hall of Famedanluu.com10 minHN2

  3. Details of the Cloudflare outage on July 2, 2019

    A Cloudflare WAF Managed Rule introduced a regex with catastrophic backtracking, exhausting CPU on every core and causing a 27‑minute global outage. The team used synthetic tests, CPU metrics, and a global termination mechanism to halt the rule, then rolled it back after confirming the fix.

    Hall of Famecloudflare.com22 minpostmortemHN698149

  4. New low-cost burstable Amazon EC2 T8i instances are generally available

    AWS announced general availability of T8i burstable EC2 instances, which use 6th‑gen Intel Xeon and Nitro to deliver up to 30% better price performance and up to 70% higher compute than T3. They come in four tiny sizes, keep the same CPU‑credit model, and target low‑to‑moderate workloads.

    AWSamazon.com3 minrelease

  5. Metastable Failures in Distributed Systems

    This paper introduces and formalizes "metastable failures" in distributed systems, a class of outages where a trigger pushes a system into a bad state that persists due to a sustaining effect, even after the trigger is removed. These failures often stem from features designed for efficiency or reliability and require significant external intervention to resolve.

    Hall of Famesigops.org25 minpaperHN16112

  6. September 16, 2026 security incident: how we responded to a LiteSpeed zero-day attack

    Hostinger detected a zero‑day exploit in LiteSpeed Web Server on a Brazil server on Sep 16 2026, giving the attacker root access and compromising 399 accounts. The team isolated the server, coordinated with LiteSpeed to patch the vulnerability, rolled out the fix fleet‑wide, and restored affected sites from backups within the same day.

    Hostingerhostinger.com2 minpostmortem