TailscaleAlex Chan14 min readpostmortemintermediate
How we tracked down a 16-year-old SQLite bug
Summary
Tailscale experienced 19 SQLite database corruptions over six months due to a 16‑year‑old bug in SQLite’s WAL checkpoint logic. The single‑writer, Go‑driven shard architecture forced them to add forensic telemetry, a transaction‑logging replay pipeline, and work directly with SQLite core developers to isolate and fix the issue, dramatically reducing downtime.
- Even well‑tested, single‑writer SQLite can corrupt under rare edge cases; production telemetry is essential for catching them.
- Streaming every mutating SQL statement to a separate log enables deterministic replay and can serve as a recovery path when backups are insufficient.
- Collaboration with upstream maintainers (SQLite pro‑support) can accelerate root‑cause analysis for obscure bugs.
- Automated integrity checks on backups and hard‑stop shutdown on detection cut mean‑time‑to‑recovery from >1 h to <1 h.
The post shows a realistic, large‑scale failure mode in a widely‑used embedded database and demonstrates practical forensics and mitigation strategies that other services with SQLite (or similar WAL‑based stores) can adopt.
7/10
