proomt

Search

Search posts, papers, and topics

Hall of Fame

Hall of FameSanjay Ghemawat, Howard Gobioff, Shun-Tak Leung200362 min readpaperintermediate

The Google File System

Read on google.com

Summary

The Google File System (GFS) is a scalable distributed file system designed for Google's data-intensive applications, built on inexpensive commodity hardware. It provides fault tolerance and high aggregate performance by optimizing for large files, sequential appends, and anticipating frequent component failures.

  • GFS uses a single master for metadata and multiple chunkservers for data storage, with clients interacting directly with chunkservers for data I/O.
  • Files are divided into fixed-size chunks (e.g., 64MB), which are replicated across chunkservers for reliability and availability.
  • The system is optimized for large streaming reads and sequential appends to huge files, rather than small random writes or low-latency operations.
  • GFS assumes component failures are routine, integrating constant monitoring, error detection, and automatic recovery into its core design.

This paper is foundational for understanding modern distributed storage systems and significantly influenced the design of many subsequent systems, including Hadoop HDFS.

9/10
storagearchitecturedistributed systemsgooglefault tolerancefile system

Related reading

  1. The Design and Implementation of a Log-Structured File System

    The paper introduces a log‑structured file system (LFS) that writes all data sequentially to a log and uses a segment cleaner to reclaim space. In the Sprite LFS prototype, write throughput reaches 65‑75 % of raw disk bandwidth, an order of magnitude faster than Unix for small files, while reads remain comparable.

    Hall of Fameberkeley.edu53 minpaper

  2. Introducing Filestore agent volumes: fully managed storage for agent workspaces

    Google Cloud adds Filestore agent volumes, a fully‑managed, elastic file‑system that automatically provisions isolated POSIX workspaces for GKE‑based AI agent sandboxes. Volumes attach in milliseconds, support RWX with file‑level locking, and charge only for used capacity with automatic tiering, aiming to cut cold‑start latency and storage waste for large‑scale agent fleets.

    Google Cloud Bloggoogle.com4 min

  3. Bigtable: A Distributed Storage System for Structured Data

    Bigtable is a distributed storage system for structured data, designed to scale to petabytes across thousands of commodity servers. It provides a sparse, distributed, persistent multidimensional sorted map indexed by row, column, and timestamp, used by many Google products.

    Hall of Famegoogle.com48 minpaper

  4. The Chubby Lock Service for Loosely-Coupled Distributed Systems

    Chubby is Google's lock service for coarse-grained synchronization and reliable low-volume storage in loosely-coupled distributed systems, built on Paxos. It provides a file-system-like interface with advisory locks and event notifications, and its design evolved significantly from initial expectations based on real-world use.

    Hall of Famegoogle.com59 minpaper

  5. Spanner: Google's Globally-Distributed Database

    Spanner is Google's globally-distributed, synchronously-replicated database, notable for being the first to support externally-consistent distributed transactions at global scale. Its key innovation is the TrueTime API, which exposes clock uncertainty to enable these strong consistency guarantees.

    Hall of Famegoogle.com46 minpaper