proomt

Search

Search posts, papers, and topics

Hall of Fame

Hall of FameDan Luu201522 min readintermediate

Files are hard

Read on danluu.comHN 5·1 comments

Summary

The post explains why desktop file‑based apps often corrupt data: filesystem write ordering and crash‑consistency are subtle, and default journaling modes don’t guarantee atomic updates without explicit fsync barriers.

  • Use an undo‑log plus fsync on the log, the data file, and the parent directory to get crash‑consistent updates on ext3/ext4 regardless of data mode.
  • The default ext3 "ordered" mode can reorder writes; you must add explicit fsync calls to enforce ordering.
  • Filesystem semantics differ (ext3, ext4, btrfs, XFS); assuming atomicity or ordering leads to bugs in many apps (git, LevelDB, HDFS).
  • On macOS you need fcntl(F_FULLFSYNC) for a true flush; Linux fsync may be a no‑op on some disks or kernels.

Anyone building reliable storage‑oriented software or desktop apps must understand filesystem ordering and crash‑consistency to avoid silent data corruption.

6/10
storagelinuxreliabilityfilesystemscrash consistency

Related reading

  1. Plain-text files are at risk

    The article argues that plain‑text files, once the universal data interface, are losing relevance as smartphones lack native editors and developers shift to LLM‑driven REPLs, risking future loss of support for plain‑text tools.

    Lobsterssr.ht3 minlobste.rs37

  2. How we tracked down a 16-year-old SQLite bug

    Tailscale experienced 19 SQLite database corruptions over six months due to a 16‑year‑old bug in SQLite’s WAL checkpoint logic. The single‑writer, Go‑driven shard architecture forced them to add forensic telemetry, a transaction‑logging replay pipeline, and work directly with SQLite core developers to isolate and fix the issue, dramatically reducing downtime.

    Tailscaletailscale.com14 minpostmortemHN1223239

  3. Put App Documents in the System File Browser

    Codename One adds a read‑only DocumentProvider API that lets apps publish a virtual file tree to iOS Files and Android’s storage picker. The tree is defined with `DocumentNode` objects, can include remote‑only entries, and is shared via an App Group container. The iOS extension runs in a separate process, so the model is serialized and read independently; Android uses the same model inside the ap…

    CodeName Onecodenameone.com4 min

  4. The Design and Implementation of a Log-Structured File System

    The paper introduces a log‑structured file system (LFS) that writes all data sequentially to a log and uses a segment cleaner to reclaim space. In the Sprite LFS prototype, write throughput reaches 65‑75 % of raw disk bandwidth, an order of magnitude faster than Unix for small files, while reads remain comparable.

    Hall of Fameberkeley.edu53 minpaper

  5. Reinventing issue tracking: Local-first and Git-native

    A devlog describing how Manganin’s issue tracker stores issues in a dedicated Git repo as plain files, after rejecting earlier approaches that tried to embed issue data in the code tree or Git refs. The author explains the pitfalls of using refs for issue storage, shows the low‑level Git commands needed for that approach, and then outlines the final design: a hidden sister repository where each i…

    Lobstersmanganin.dev6 minHN1lobste.rs27

  6. App Hardening: One Obfuscation Pipeline Across Every Port

    Codename One adds a cloud‑side hardening step that runs on the merged JAR before it is split into Android, iOS, JavaScript, and desktop binaries. It can rename symbols, encrypt string literals, and insert opaque‑predicate control‑flow guards at configurable levels (off → standard → aggressive → paranoid). The transforms are selective per platform to avoid breaking optimizers, and a mapping is kep…

    CodeName Onecodenameone.com6 min