Hacker News front page18 min readintermediate
You can run Git on object storage if you re-make packfiles
Summary
The author built a Git server on object storage (objgit) and found standard Git packfiles inefficient due to random‑access and round‑trip latency. By designing a columnar, object‑storage‑native packfile format that can be accessed via HTTP range requests, they achieved acceptable performance for large repositories without client‑side changes.
- Git stores objects as loose files or in packfiles; the latter are optimized for local filesystems with mmap, not for remote object storage.
- Fetching millions of objects from object storage (≈10 ms per GET) would take hours, so bundling them is essential.
- Standard packfile indexes give object offsets but not compressed sizes, making precise HTTP Range requests impossible.
- A custom packfile format that records both offset and compressed length enables efficient range reads from object storage.
Running Git directly on object storage can simplify infrastructure and reduce costs, but only if the storage access patterns are optimized. This post shows a concrete redesign of Git’s packfile layout to make remote object stores viable for large codebases.
5/10
