proomt

Search

Search posts, papers, and topics

Hall of Fame

Hall of FameMendel Rosenblum, John K. Ousterhout199153 min readpaperadvanced

The Design and Implementation of a Log-Structured File System

Summary

The paper introduces a log‑structured file system (LFS) that writes all data sequentially to a log and uses a segment cleaner to reclaim space. In the Sprite LFS prototype, write throughput reaches 65‑75 % of raw disk bandwidth, an order of magnitude faster than Unix for small files, while reads remain comparable.

  • LFS stores all file data and metadata in a single on‑disk log, eliminating random writes and seeks.
  • Segments are cleaned by a cost‑benefit algorithm that separates hot (young) and cold (old) data.
  • Crash recovery only scans the tail of the log, making it much faster than traditional file‑system scans.
  • Sprite LFS achieves ~70 % of disk bandwidth for writes versus 5‑10 % for Unix FFS, especially for small files.

File‑system engineers and storage researchers should study LFS because its log‑based design and cleaning policy underpin many modern systems such as SSD write‑amplification mitigation and log‑structured merge trees.

9/10

Related reading

  1. 1 points

    Saving another 100TB of RAM with math (and Rust)

    Cloudflare reduced the memory footprint of its Pingora Backend Router by re‑examining the consistent‑hashing implementation in the pingora‑ketama library. By increasing the number of virtual hash points per server from the default 1 to the standard 160 (and applying weighted hashing based on disk capacity), they cut the per‑node overhead enough to reclaim >100 TB of RAM across the fleet. The post…

    Hacker News front pagecloudflare.com13 minHN478120lobste.rs33
  2. The UNIX Time-Sharing System

    The 1974 paper outlines the original UNIX operating system for the PDP‑11, detailing its hierarchical file system, device file abstraction, and rewrite in C. It demonstrates how a small, self‑contained OS could provide multi‑user time‑sharing with a simple, uniform I/O model.

    Hall of Fameberkeley.edu43 minpaperHN71
  3. We're making Tailscale faster

    Tailscale is cutting memory overhead for small packets, adding a multi‑queue pipeline for routers and exit nodes, using Linux’s writev, and introducing netmap caching to speed up startup. These changes give ~5 % throughput gains now and larger gains in upcoming releases.

    Tailscaletailscale.com7 minHN9540
  4. 6.3x Faster Testing with Retroactive Logging

    Antithesis developed "retroactive logging" to address the performance overhead of verbose debugging logs during deterministic fuzz testing. By initially sinking most logs within the hypervisor and replaying runs with full logging only on failure, they achieved 6.3x faster testing.

    Antithesisantithesis.com4 min