proomt

Search

Search posts, papers, and topics

Hall of Fame

Hall of FameJames Hamilton200743 min readpaperintermediate

On Designing and Deploying Internet-Scale Services

Read on usenix.org

Summary

James Hamilton’s 2007 paper distills a set of concrete best‑practice tenets for building operations‑friendly, internet‑scale services—design for failure, keep things simple, and automate everything. Applying these guidelines (commodity hardware, single‑version software, redundancy, multi‑tenancy) enables ratios of thousands of servers per administrator while maintaining reliability.

  • Design services to survive any component failure without human intervention; test failure paths by forcing hard failures.
  • Use commodity hardware and single‑version deployments to simplify automation and reduce operational cost.
  • Build synchronous redundancy so any server can be taken down without draining workload, ensuring high availability.
  • Keep the service model simple, avoid unnecessary dependencies, and employ multi‑tenancy for efficient resource use.

SREs and architects building large‑scale, customer‑facing services should care because the paper distills proven practices for achieving high system‑to‑admin ratios and reliable automation.

6/10
automationarchitecturedistributed systemsscalabilityreliabilityoperations

Related reading

  1. The Tail at Scale

    The paper defines “tail‑tolerant” systems that keep interactive latency (≤100 ms) low even as services scale to thousands of servers, and surveys the main causes of latency outliers. It shows that techniques such as request replication, hedged requests, and resource isolation can dramatically shrink the latency tail with modest overhead, enabling higher utilization without over‑provisioning.

    Hall of Fameresearch.google1 minpaper

  2. How Uber Protects Against Retry Storms

    Uber developed a context-aware mechanism to prevent retry storms in deep microservice dependency chains. It introduces "error ownership" where services claim errors they originate and unclaim errors they propagate, allowing upstream callers to make informed retry decisions and avoid amplifying load on already struggling services.

    Hacker News front pageuber.com12 minHN11949

  4. Article: Beyond Relevance: A Governance-First Architecture for Enterprise Personalization

    The article proposes a governance‑first architecture for enterprise personalization, where policy‑driven steps (memory, journey graph, AI routing, scoring, trust checks, outcome simulation) shape the recommendation before it is returned. A reference FastAPI implementation demonstrates the pattern with external YAML policies and optional LLM assistance.

    InfoQinfoq.com19 min

  5. Shared Selective Persistent Memory for Agentic LLM Systems

    Apple proposes a memory architecture for agentic LLMs that selectively persists reusable context (specs, schemas, configs, constraints) across sessions and users. Shared workspaces with role‑based access and a zero‑token data‑refresh mechanism cut token usage by 97×, reduce task time by 14×, and raise task‑completion rates to 96% versus 71%‑79% for baselines.

    Apple Machine Learning Researchapple.com1 minpaper