proomt

Search

Search posts, papers, and topics

Hall of Fame

Hall of FameBrendan Burns, Brian Grant, David Oppenheimer, Eric Brewer, John Wilkes201625 min readpaperintermediate

Borg, Omega, and Kubernetes

Read on googleusercontent.comHN 2·0 comments

Summary

Google developed three container management systems: Borg, Omega, and Kubernetes, over a decade. This article details their evolution, design principles, and the lessons learned from managing containers at scale, emphasizing the shift from machine-oriented to application-oriented infrastructure.

  • Borg unified long-running services and batch jobs, using Linux containers for resource sharing and cost reduction.
  • Omega improved Borg's ecosystem with a more consistent, principled architecture, using a Paxos-based store and optimistic concurrency.
  • Kubernetes, open-source, focused on developer experience and exposed state via a domain-specific REST API, unlike Omega's direct store access.
  • Containers enable higher utilization by colocating diverse workloads and abstracting away machine/OS details for application developers.

Engineers working with container orchestration systems like Kubernetes should read this to understand the foundational design decisions and historical context from Google's pioneering work.

9/10
devopscontainersarchitecturedistributed systemskubernetesgoogle

Related reading

  1. Large-scale cluster management at Google with Borg

    Google's Borg is a large-scale cluster manager running hundreds of thousands of jobs across tens of thousands of machines. It achieves high utilization through efficient task-packing and over-commitment, while ensuring high availability and simplifying resource management for users.

    Hall of Famegoogle.com60 minpaper

  2. Kubernetes v1.37: Pod-Level Resource Managers graduated to Beta

    Kubernetes v1.37 adds Pod‑Level Resource Managers to beta (off by default). The feature lets Kubelet’s Topology, CPU, and Memory managers consume pod‑level `.spec.resources` to reserve exclusive NUMA‑aligned CPUs/memory for primary containers while sidecars share a pod‑isolated pool. A new PodResources gRPC API now reports `cpu_ids` and `memory` per pod. Enable via the `PodLevelResourceManagers`…

    Kuberneteskubernetes.io2 min

  4. Kubernetes 1.37 Released: Stable Metrics API and Rootless Kubelet in Beta

    Kubernetes 1.37 (Garhwal) ships GA Metrics API, beta rootless kubelet, GA resilient watchcache, GA pod certificates, and a suite of alpha/beta features (workload‑aware scheduling, pod checkpoint/restore, StatefulSet Recreate strategy). The release focuses on stability, security, and AI/ML cost‑optimisation.

    InfoQinfoq.com2 minrelease

  5. Introducing Filestore agent volumes: fully managed storage for agent workspaces

    Google Cloud adds Filestore agent volumes, a fully‑managed, elastic file‑system that automatically provisions isolated POSIX workspaces for GKE‑based AI agent sandboxes. Volumes attach in milliseconds, support RWX with file‑level locking, and charge only for used capacity with automatic tiering, aiming to cut cold‑start latency and storage waste for large‑scale agent fleets.

    Google Cloud Bloggoogle.com4 min

  6. Google Open-Sources AX a Kubernetes Style Orchestrator for Autonomous AI Agents

    Google open‑sources AX, a Kubernetes‑style orchestrator for autonomous AI agents that treats each agent as a stateful actor, checkpointing idle sessions and resuming them in sub‑second latency. AX introduces four CRDs (Task, Workspace, Gateway, Model) and a Go CLI, runs on top of the Agent Substrate runtime, and uses gVisor sandboxes with Redis‑backed control plane. The design targets the bursty,…

    InfoQinfoq.com3 min