Bots - New options to manage AI traffic

<p>Not all AI traffic is the same. Now, all customers — including those on the Free plan — can manage AI crawlers based on what they actually do on your site. Cloudflare groups AI traffic into three behaviors you can control independently: <a href="https://developers.cloudflare.com/bots/concepts/bot/#ai-bots">Search, Agent, and Training</a>. This lets you keep the automated traffic that sends readers and revenue back to you, while blocking the traffic that only takes from your content.</p> <p>Each behavior maps to a real use case. <strong>Search</strong> covers crawlers that index your content so they can answer questions about it later, where you should expect referral traffic or other equitable compensation in return. <strong>Agent</strong> covers automated activity acting in real time on a person's behalf, such as chat fetch bots and browser-use agents. <strong>Training</strong> covers crawlers that take your content to train or fine-tune a model. For each preset you can choose to block on all pages, block only on pages that display ads, or choose not to block.</p> <img src="https://developers.cloudflare.com/cdn-cgi/image/onerror=redirect,width=4720,height=2966,format=webp/_astro/ai-bot-traffic-policies.BqXU7Gmv.png" alt="The Configure AI bot traffic policies screen, where Search, Agent, and Training can each be set to allow, block, or block only on pages with ads" loading="lazy" decoding="async" width="4720" height="2966"> <p>Starting <strong>September 15, 2026</strong>, new domains onboarding to Cloudflare receive updated defaults: Bots classified as Training or as Agent are blocked on pages that display ads, while <strong>Search</strong> remains allowed. On that date, multi-purpose crawlers that combine Search and Training will be affected by the new defaults to block Training. All customers can <a href="https://dash.cloudflare.com/?to=/:account/:zone/security/settings" target="_blank" rel="noopener">opt out of the new defaults<span class="external-link"> ↗︎</span></a> at any time before September 15.</p>