proomt

Search

Search posts, papers, and topics

Hall of Fame

Hall of FameGitLab201717 min readpostmortemadvanced

Postmortem of database outage of January 31

Read on gitlab.com

Summary

GitLab.com suffered a major outage and data loss when an engineer accidentally wiped the primary PostgreSQL database during a replication resync attempt. Multiple backup and recovery procedures failed, including `pg_dump` due to a version mismatch and silent cronjob failures, leading to significant data loss.

  • An engineer accidentally ran `rm -rf` on the primary PostgreSQL server instead of the secondary during a replication resync.
  • `pg_dump` backups failed silently for months due to a PostgreSQL version mismatch (9.2 vs 9.6) and DMARC blocking cronjob emails.
  • `pg_basebackup`'s silent hanging behavior during replication setup contributed to engineer confusion.
  • LVM snapshots, intended for staging, were the only viable recovery option, resulting in 6-7 hours of data loss.

This postmortem offers a candid and detailed account of multiple cascading failures in operations, backups, and documentation, providing crucial lessons for any team managing critical infrastructure.

8/10
databasespostgresdisaster recoverypostmortemoperationsbackups

Related reading

  1. Reading postmortems

    Dan Luu surveys public postmortems and finds that most severe outages stem from a handful of recurring causes—poor error‑handling code, risky configuration changes, hardware faults, manual processes, and missing monitoring/alerting. He backs the claims with study numbers and argues engineers should focus on testing, automation, and observability to cut these failure modes.

    Hall of Famedanluu.com10 minHN2

  3. How we tracked down a 16-year-old SQLite bug

    Tailscale experienced 19 SQLite database corruptions over six months due to a 16‑year‑old bug in SQLite’s WAL checkpoint logic. The single‑writer, Go‑driven shard architecture forced them to add forensic telemetry, a transaction‑logging replay pipeline, and work directly with SQLite core developers to isolate and fix the issue, dramatically reducing downtime.

    Tailscaletailscale.com14 minpostmortemHN1223239

  4. Introducing Neon Labs

    Neon Labs is a new experimental web app (Next.js + TypeScript) that offers two PostgreSQL upgrade tools: a live‑catalog Upgrade Assessment that flags version‑specific blockers, and a Migration Assistant that auto‑generates import, pg_dump/restore, or logical‑replication pipelines based on DB size and downtime needs. The tools run read‑only against a Neon project, never expose connection strings,…

    Neonneon.com6 minrelease

  5. Database Branching: A Developer's Guide to Git-Style Workflows

    Database branching uses copy‑on‑write to give developers, CI jobs, and AI agents isolated database snapshots without full copies. Branches share unchanged data, store only deltas, and are disposable, enabling production‑like testing, per‑PR isolation, safe experimentation, and rapid cleanup. Safe operation requires protecting parent branches, using mock data, TTLs, and treating migrations as the…

    Databricksdatabricks.com9 min

  6. Postgres on NVMe: performance and the convergence of transactions and analytics

    Local NVMe storage cuts Postgres I/O latency from ms to µs, yielding ~9× higher TPS and 10× lower transaction latency on a 482 GiB pgbench workload. The gain comes from reduced IO wait, not more CPU work. To retain durability, combine NVMe with quorum synchronous replication and continuous WAL archiving (WAL‑G). For analytics, offload scans to ClickHouse via WAL‑based CDC (pg_clickhouse or the ne…

    ClickHouseclickhouse.com8 min