TemporalChandler Ortman9 min readintermediate
Running out of runway: Migrating Temporal Cloud's billing store to ClickHouse
Summary
Temporal Cloud migrated its billing store to ClickHouse, gaining 3-5 years of headroom and reducing costs by 70%. The migration focused heavily on a six-week shadowing period to validate data correctness, which uncovered an integer overflow bug in ClickHouse for large accounts.
- Shadowing critical data migrations with background comparisons is crucial for correctness.
- Define expected vs. blocking differences upfront, including float precision and freshness gaps.
- Test shadowing on high-volume accounts first to expose scale-related bugs like integer overflows.
- Roll out critical data path changes in order of reversibility, starting with read-only APIs.
Engineers managing critical data migrations, especially for billing or financial systems, will find valuable lessons on validation strategies and thoughtful rollout methodologies.
7/10
