proomt

Search

Search posts, papers, and topics

Hall of Fame

Hall of FameJay Kreps201354 min readintermediate

The Log: What every software engineer should know about real-time data's unifying abstraction

Read on archive.org

Summary

The log is a simple, append-only, totally-ordered sequence of records, fundamental to distributed systems and databases. It serves as a unifying abstraction for data integration, real-time processing, and building consistent systems by providing a shared, ordered history of events.

  • A log is an append-only, totally-ordered sequence of records, indexed by a unique sequential number.
  • Logs are central to database ACID properties, ensuring durability and enabling replication via log shipping.
  • The State Machine Replication Principle uses logs to feed identical, deterministic processes the same ordered inputs for consistency.
  • Logs provide a robust mechanism for data integration, real-time processing, and building fault-tolerant distributed systems.

Engineers building or operating distributed data systems should understand logs as they are the foundational abstraction for consistency, replication, and fault tolerance.

9/10
architecturedata engineeringdistributed systemsdatabasesreplicationconsensus

Related reading

  1. The Log-Structured Merge-Tree (LSM-Tree)

    The Log-Structured Merge-Tree (LSM-tree) is a disk-based data structure designed to provide low-cost indexing for files with high insert rates, like history tables or logs. It defers and batches index changes, cascading them from memory to disk components, significantly reducing disk I/O compared to B-trees for writes.

    Hall of Fameumb.edu88 minpaper

  3. The Design and Implementation of a Log-Structured File System

    The paper introduces a log‑structured file system (LFS) that writes all data sequentially to a log and uses a segment cleaner to reclaim space. In the Sprite LFS prototype, write throughput reaches 65‑75 % of raw disk bandwidth, an order of magnitude faster than Unix for small files, while reads remain comparable.

    Hall of Fameberkeley.edu53 minpaper

  5. CISA's Logging Reference Architecture for OMB M-26-14: What federal agencies should do next

    CISA’s Logging Reference Architecture translates OMB M‑26‑14 requirements into eight concrete shifts, from searchable storage tiers to schema‑first design and continuous validation. Agencies should use Elastic’s tools to build a heterogeneous, schema‑first logging architecture and file a plan by 18 Nov 2026 to meet the mandated maturity milestones.

    Elasticelastic.co11 min

  6. ARIES: A Transaction Recovery Method Supporting Fine-Granularity Locking and Partial Rollbacks Using Write-Ahead Logging

    ARIES is a transaction recovery method using write-ahead logging (WAL) that supports fine-granularity locking and partial rollbacks. It introduces the "repeating history" paradigm to redo all missing updates before performing rollbacks of loser transactions during system restart, using Log Sequence Numbers (LSNs) on pages.

    Hall of Famestanford.edu155 minpaper