proomt

Search

Search posts, papers, and topics

All posts

DatabricksHongwen (Olivia) Song7 min readintermediate

RADAR: Catch gray failures with anomaly detection

Read on databricks.com

Summary

Databricks built RADAR, a four‑stage, metric‑agnostic pipeline that uses streaming anomaly detection (SPOT) to surface gray failures in minutes with >90% precision. The blog shows how to recreate the system on Databricks for any metric, from billing to model drift.

  • Gray failures are partial, silent incidents that evade traditional health checks; real‑time anomaly detection catches them early.
  • RADAR’s pipeline: reliability metrics → SPOT streaming anomaly detection → enriched, deduped alerts → AI‑assisted root‑cause dashboards.
  • The system runs on native Databricks components (Zerobus, Delta Lake, MLflow, SQL Alerts, AI/BI Genie) and can be deployed via a Declarative Asset Bundle.
  • Using a single metric and a short AI prompt, you can auto‑generate the full RADAR stack for any business‑critical signal.

SREs and platform engineers need automated detection of partial failures to reduce incident discovery time and protect revenue.

6/10
devopsobservabilitydata engineeringanomaly detection

Related reading

  1. What is AIOps?

    Databricks’ blog post explains what AIOps is, its core components (data ingestion, normalization, anomaly detection, correlation, RCA, automation, collaboration), and why it’s gaining traction now. It positions AIOps as a layer between observability and action, emphasizing human‑in‑the‑loop for high‑risk steps, and outlines domain‑centric vs. domain‑agnostic approaches and common use‑cases like R…

    Databricksdatabricks.com13 min

  2. How energy teams turn theft detection into governed action with Genie and AI business processes

    Databricks shows how to turn energy‑theft ML scores into a governed, end‑to‑end workflow using a Databricks App, Lakebase for live case state, Unity Catalog for data governance, and Genie One for natural‑language executive reporting. The pattern lets utilities act on alerts faster while staying compliant, and can be reused for other fraud‑type use cases.

    Databricksdatabricks.com6 min

  4. How Databricks’ marketers use data 3x more with Genie, an AI analytics assistant

    Databricks built Marge, a Genie‑powered conversational analytics assistant on a governed Marketing Lakehouse. By starting with a single high‑value use case (email campaign performance), documenting data, encoding verified answers, teaching business terminology, and embedding the tool in existing ticket workflows, they achieved 85% adoption, 3× higher data usage in decisions, 50% QoQ usage growth,…

    Databricksdatabricks.com10 min

  5. Database for AI Agents: 5 Evaluation Criteria

    Databricks outlines five criteria for a production‑ready database for AI agents—branch‑per‑agent isolation, serverless scale‑to‑zero, hybrid search, ACID guarantees, and a unified platform that eliminates ETL lag—illustrating each with features of its Lakebase offering and brief customer anecdotes.

    Databricksdatabricks.com10 min

  6. "Regex for Rows": Simplifying Pattern Detection in SQL with MATCH_RECOGNIZE

    Databricks announces MATCH_RECOGNIZE, a preview‑only SQL clause that lets you write regex‑style patterns over ordered rows. The post walks through four industry‑level use cases (security login‑failure sequences, V‑shaped stock trends, e‑commerce cart abandonment, IoT sensor‑driven failure prediction) and claims the operator replaces complex CTEs, window functions, and self‑joins. No actual syntax…

    Databricksdatabricks.com5 min