Hacker News front page12 min readintermediate
How Uber Protects Against Retry Storms
Summary
Uber developed a context-aware mechanism to prevent retry storms in deep microservice dependency chains. It introduces "error ownership" where services claim errors they originate and unclaim errors they propagate, allowing upstream callers to make informed retry decisions and avoid amplifying load on already struggling services.
- Retry storms amplify load exponentially in deep call chains, even with retry budgets.
- Uber's solution establishes "error ownership" to differentiate between errors a service causes and those it merely propagates.
- Error ownership is determined by correlating inbound and outbound failures using a Service Dependency Analysis Solution.
- Callers only retry if the downstream service explicitly claims the error, preventing unnecessary retries for propagated errors.
Engineers operating large-scale microservice architectures with deep dependency graphs should care about this approach to prevent cascading failures and improve system reliability during partial outages.
7/10
