proomt

Search

Search posts, papers, and topics

Hall of Fame

Hall of FameJeffrey Dean, Luiz André Barroso201321 min readpaperadvanced

The Tail at Scale

Read on barroso.org

Summary

The paper explains why rare latency outliers dominate overall response time in large‑scale services and shows how tail latency grows with fan‑out. It presents practical techniques—service class prioritization, synchronized background work, and hedged requests—to keep the tail short with modest overhead.

  • Fan‑out amplifies tail latency: even a 0.1% 99th‑percentile per node can cause >60% of requests to exceed deadlines when querying 100 nodes.
  • Hedged requests send a duplicate to another replica after a short delay, cutting tail latency with low extra cost.
  • Prioritize interactive traffic via service classes and shallow queues to avoid head‑of‑line blocking.
  • Synchronize background activities across machines to confine latency spikes to brief periods rather than continuous tail impact.

Engineers building large‑scale interactive services need to understand and mitigate tail latency to deliver responsive user experiences.

8/10
performancedistributed systemsreplicationlatencytail latency

Related reading

  1. The Tail at Scale

    The paper defines “tail‑tolerant” systems that keep interactive latency (≤100 ms) low even as services scale to thousands of servers, and surveys the main causes of latency outliers. It shows that techniques such as request replication, hedged requests, and resource isolation can dramatically shrink the latency tail with modest overhead, enabling higher utilization without over‑provisioning.

    Hall of Fameresearch.google1 minpaperHN2

  2. We're making Tailscale faster

    Tailscale is cutting memory overhead for small packets, adding a multi‑queue pipeline for routers and exit nodes, using Linux’s writev, and introducing netmap caching to speed up startup. These changes give ~5 % throughput gains now and larger gains in upcoming releases.

    Tailscaletailscale.com7 minHN9540

  3. Faster Maps: Chasing Swiss Speed

    ParparVM’s HashMap suffered catastrophic miss latency due to linear probing on dense integer keys. By adopting CPython‑style perturbed probing (Swiss‑table style) and extending tagged immediate values to more primitives, miss latency dropped from 32 s to ~45 ms, allocation pressure fell dramatically, and overall performance stayed roughly flat despite a modest hit‑time slowdown.

    CodeName Onecodenameone.com8 min

  5. On Designing and Deploying Internet-Scale Services

    James Hamilton’s 2007 paper distills a set of concrete best‑practice tenets for building operations‑friendly, internet‑scale services—design for failure, keep things simple, and automate everything. Applying these guidelines (commodity hardware, single‑version software, redundancy, multi‑tenancy) enables ratios of thousands of servers per administrator while maintaining reliability.

    Hall of Fameusenix.org43 minpaper

  6. Server Monitoring in the age of AI: What static thresholds miss and how adaptive monitoring fixes it?

    Static CPU/memory thresholds generate noise because workloads vary by time‑of‑day, day‑of‑week, and long‑term trends. Adaptive monitoring learns per‑server baselines (using simple ML on historic metrics) and creates dynamic thresholds plus anomaly alerts. ManageEngine OpManager’s Zia engine is presented as a turnkey AIOps solution that auto‑learns baselines, lets you set sensitivity, and adds ale…

    SitePointsitepoint.com6 min