Search posts, papers, and topics
Uber developed a context-aware mechanism to prevent retry storms in deep microservice dependency chains. It introduces "error ownership" where services claim errors they originate and unclaim errors they propagate, allowing upstream callers to make informed retry decisions and avoid amplifying load on already struggling services.
Spotify’s AI‑assisted development doubled change volume, exposing gaps in alerting, capacity planning, fleet‑update safety, and mobile quality signals. The team added end‑to‑end monitoring, priority‑based tiering, stronger rollback/observability, and expanded edge capacity. Data shows AI‑generated code isn’t a direct incident cause, but verification pipelines must scale with velocity.
Production AI agents often fail silently due to issues beyond model hallucination, such as state inconsistencies. This talk outlines principles for building reliable agent harnesses, focusing on explicit state ownership, ordered mutations, and verifiable actions, using real-world case studies.
Canva added a lightweight, local backpressure loop to its queue worker library that monitors per‑message success/failure, computes a backoff factor against a configurable failure‑rate set‑point, and throttles the worker’s concurrency. In two real incidents the mechanism kept failure rates under 2 % fleet‑wide, limited DLQ growth to a handful of messages, and maintained throughput without manual i…
ModaLens introduces a paired image-swap audit to measure how radiology report availability affects image sensitivity in medical VLMs. It found that MedGemma-27B's answers changed significantly more often when the image was swapped if the report was not available, indicating reports reduce image reliance.
The article outlines a structured Verify→Investigate→Report→Resolve→Review workflow for GCP reliability incidents and stresses pre‑incident preparation across design, data, playbooks, and training. It lists concrete tools (Cloud Logging, Service Health, Gemini Assist) and reporting steps to help engineers reduce outage impact.
This Jepsen blog post announces a retrospective talk on 13 years of testing distributed systems, linking to slides and a video. The post itself contains no detailed lessons or technical content.
This article outlines essential API design considerations, covering HTTP fundamentals, architectural styles like REST and GraphQL, and critical aspects such as naming, versioning, security, and reliability. It serves as a high-level checklist for engineers designing or consuming APIs.
