Hall of FameBo Ingram202311 min readintermediate
How Discord Stores Trillions of Messages
Summary
Discord migrated its trillion‑message store from a problematic Cassandra cluster to ScyllaDB, adding Rust‑based data services that coalesce requests and hash‑route traffic to tame hot partitions and reduce latency.
- Cassandra hot partitions caused cluster‑wide latency spikes because reads must scan many SSTables; writes are cheap but reads dominate traffic.
- ScyllaDB removed Java GC pauses and offered shard‑per‑core isolation, but hot partitions still required upstream mitigation.
- Rust data services expose a gRPC per query, coalesce concurrent reads, and use consistent‑hash routing so identical channel requests hit the same service instance.
- Migration used dual‑write, Spark migrator, then a custom Rust migrator that checkpoints token ranges in SQLite for zero‑downtime bulk transfer.
Teams operating large, write‑heavy message stores need practical patterns for handling hot partitions and zero‑downtime migrations.
7/10
