proomt

Search

Search posts, papers, and topics

All posts

InfoQMark Silvester3 min readintermediate

Lyft Moves Streaming Fleet to Apache Flink Kubernetes Operator

Read on infoq.com

Summary

Lyft migrated its hundreds of production Flink jobs from a home‑grown Kubernetes operator to the Apache Flink Kubernetes Operator, gaining last‑state upgrades, in‑place autoscaling, and resource autotuning while cutting typical deployment downtime to 3–6 minutes. The switch also enabled Flink 1.19 features, saved millions in over‑provisioned capacity, and required workload‑specific scaling strate…

  • Moving to the Apache Flink Kubernetes Operator enabled last-state upgrades, in-place autoscaling, and resource autotuning, reducing typical deployment downtime to 3–6 minutes (20 minutes for the largest jobs).
  • The operator's built-in last-state mode restores from HA metadata or latest checkpoint, avoiding the legacy operator's savepoint retry and idempotency problems.
  • Upgrading to Flink 1.19 unlocked in-place scaling and KinesisStreamsSource metrics, allowing the autoscaler to right-size the fleet and save millions of dollars per year.
  • Autotuning that resizes container memory requires pod restarts, so Lyft split workloads: critical jobs use in-place autoscaling, less critical jobs accept restarts for autotuning.

Engineers running streaming workloads on Kubernetes need to understand the operational benefits and trade‑offs of adopting the Apache Flink operator versus a custom solution.

6/10
streamingkubernetesflinkautoscalingoperator

Related reading

  1. CloudBees vs Harness: Why Migration Isn't the Fix

    The article argues that Harness’s “free migration” offer hides significant downstream costs (training, pipeline rebuilds, compliance recertification) and that even after migration you still lack unified governance across heterogeneous CI/CD tools. CloudBees positions its Unify control plane as a tool‑agnostic layer that adds visibility, continuous governance, AI‑driven test selection, and hybrid…

    Codeshipcloudbees.com5 min
  2. 1 points

    Saving another 100TB of RAM with math (and Rust)

    Cloudflare reduced the memory footprint of its Pingora Backend Router by re‑examining the consistent‑hashing implementation in the pingora‑ketama library. By increasing the number of virtual hash points per server from the default 1 to the standard 160 (and applying weighted hashing based on disk capacity), they cut the per‑node overhead enough to reclaim >100 TB of RAM across the fleet. The post…

    Hacker News front pagecloudflare.com13 minHN478120lobste.rs33

  3. Kubernetes v1.37: Pod-Level Resource Managers graduated to Beta

    Kubernetes v1.37 adds Pod‑Level Resource Managers to beta (off by default). The feature lets Kubelet’s Topology, CPU, and Memory managers consume pod‑level `.spec.resources` to reserve exclusive NUMA‑aligned CPUs/memory for primary containers while sidecars share a pod‑isolated pool. A new PodResources gRPC API now reports `cpu_ids` and `memory` per pod. Enable via the `PodLevelResourceManagers`…

    Kuberneteskubernetes.io2 min

  4. OpenTelemetry everywhere: Migrating a metrics platform at scale

    Atlassian replaced its decade‑old gostatsd‑based metrics pipeline with a fully OpenTelemetry‑based stack by keeping the StatsD‑UDP contract on the client side and swapping in purpose‑built OTel Collector distributions for collection, ingest, aggregation, and forwarding. The migration was done incrementally, saved ~3.9% CPU per service, cut sidecar cost ~30% fleet‑wide, halved aggregation CPU, and…

    CNCFcncf.io6 minHN1

  5. Monitor TAS and gang scheduling for AI training in Kubernetes

    Kubernetes’ default scheduler can’t satisfy AI training’s need for low‑latency GPU interconnects and simultaneous pod start‑up. The blog explains how the open‑source Kueue job queue adds topology‑aware placement (using node labels like `topology.kubernetes.io/rack`) and how the Coscheduling plugin adds a permit phase that only binds a gang of pods when the full set is ready, preventing idle GPU r…

    Datadogdatadoghq.com19 min

  6. Kubernetes 1.37 Released: Stable Metrics API and Rootless Kubelet in Beta

    Kubernetes 1.37 (Garhwal) ships GA Metrics API, beta rootless kubelet, GA resilient watchcache, GA pod certificates, and a suite of alpha/beta features (workload‑aware scheduling, pod checkpoint/restore, StatefulSet Recreate strategy). The release focuses on stability, security, and AI/ML cost‑optimisation.

    InfoQinfoq.com2 minrelease