proomt

Search

Search posts, papers, and topics

All posts

CloudflareBryan Becker10 min readintermediate

Have it both ways: stay discoverable in search while disallowing AI training

Read on cloudflare.comHN 86·51 comments

Summary

Cloudflare adds a Disallow AI Training setting that lets sites stay indexed for search while blocking crawlers from using their content for AI model training. The feature replaces the generic Block AI Bots option with granular Search, Training, and Agent controls.

  • Disallow AI Training publishes a robots.txt directive that blocks training crawlers but keeps mixed‑use bots (Applebot, Googlebot, Bingbot) allowed for search.
  • The new granular controls (Search, Training, Agent) replace the old Block AI Bots setting, with automatic migration for existing customers.
  • Operators must meet the "Accountable" criteria—opt‑out mechanisms, URL‑level visibility, and no impact on search—to be recognized by Cloudflare.
  • Apple, Google, and Microsoft already qualify as Accountable; other major AI providers are categorized similarly for training‑only blocking.

Site owners can protect their content from being used to train AI models without losing search engine discoverability.

5/10
securityaisearchbot management

Related reading

  1. When scanners miss the attack: how Cloudflare Client-Side Security protects storefronts

    Cloudflare’s Page Shield uses a graph‑neural‑network (GNN) to model JavaScript as a syntax‑tree graph, followed by a lightweight LLM for second‑opinion triage and an ensemble of frontier models for deep analysis. This pipeline caught eight malicious payloads across four distinct affiliate‑theft and backdoor techniques that traditional scanners missed, demonstrating the need for runtime, behavior‑…

    Cloudflarecloudflare.com21 minHN2

  2. Cloudflare Introduces the Agent Development Lifecycle to Replace Traditional SDLC

    Cloudflare’s Agent Development Lifecycle (ADLC) replaces the classic SDLC with an autonomous, event‑driven workflow system that lets AI agents handle code generation, testing, deployment, and maintenance. The platform builds on Cloudflare Workflows and the new @cloudflare/ci library to spin up containers, run headless browsers, and chain steps with caching and credential support. Observability is…

    InfoQinfoq.com2 min

  4. How to upskill enterprise AI builders by using daily micro habits

    Google Cloud Consulting proposes a four‑pillar micro‑learning framework for enterprise AI upskilling: 5‑minute browser‑based exercises, pre‑configured sandboxes, daily streaks, and delivering runnable code each session. A pilot (Advent of Agents) showed >150k participants, 859k code runs, and a 31% daily return rate, suggesting short, frictionless tasks improve engagement versus traditional bootc…

    Google Cloud Bloggoogle.com3 min

  6. Introducing AI First Responder: The Next Step in CloudBees’ AI Journey

    CloudBees announced AI First Responder (AIFR), an AI‑powered incident‑triage assistant that lives in Slack and leverages the CloudBees Unify control plane and its knowledge graph. AIFR claims to summarize logs, suggest root causes, and optionally trigger GitOps remediation, all while logging actions for governance. The feature is in limited preview and available to a design‑partner program.

    Codeshipcloudbees.com2 minrelease