AI Search - AI Search makes it easier to build a search engine for your data

<p><a href="https://developers.cloudflare.com/ai-search/">AI Search</a> gets you from a data source to a working search endpoint quickly. This release adds what you need to put that endpoint in front of real users: your own domain, authentication, and one endpoint across several instances. It also adds crawling for sites without a complete sitemap, so your index covers everything you want it to find.</p> <p>Each of the following is a new option. The previous behavior is still the default, so nothing changes until you change it.</p> <div tabindex="-1" class="heading-wrapper level-h4"><h4 id="serve-search-from-your-own-domain">Serve search from your own domain</h4><a class="anchor-link" href="#serve-search-from-your-own-domain"><span aria-hidden="true" class="anchor-icon"><svg width="16" height="16" viewBox="0 0 24 24"><path fill="currentcolor" d="m12.11 15.39-3.88 3.88a2.52 2.52 0 0 1-3.5 0 2.47 2.47 0 0 1 0-3.5l3.88-3.88a1 1 0 0 0-1.42-1.42l-3.88 3.89a4.48 4.48 0 0 0 6.33 6.33l3.89-3.88a1 1 0 1 0-1.42-1.42Zm8.58-12.08a4.49 4.49 0 0 0-6.33 0l-3.89 3.88a1 1 0 0 0 1.42 1.42l3.88-3.88a2.52 2.52 0 0 1 3.5 0 2.47 2.47 0 0 1 0 3.5l-3.88 3.88a1 1 0 1 0 1.42 1.42l3.88-3.89a4.49 4.49 0 0 0 0-6.33ZM8.83 15.17a1 1 0 0 0 1.1.22 1 1 0 0 0 .32-.22l4.92-4.92a1 1 0 0 0-1.42-1.42l-4.92 4.92a1 1 0 0 0 0 1.42Z"></path></svg></span></a></div> <p>A <a href="https://developers.cloudflare.com/ai-search/configuration/retrieval/public-endpoint/">public endpoint</a> is a URL that a site or app can query directly, with no authentication in front of it. By default that URL is a generated hostname on <code>search.ai.cloudflare.com</code>. You can now serve the same endpoint from a <a href="https://developers.cloudflare.com/ai-search/configuration/retrieval/public-endpoint/custom-domains/">custom domain</a>, a hostname in a zone that you own:</p> <figure class="nb-code-figure" data-nb-lang="txt"><pre class="astro-code astro-code-themes github-light github-dark nb-shiki-c6xiwz" tabindex="0" data-language="txt" data-nb-lang="txt"><code><span class="line"><span class="nb-shiki-wvjl67">https://search.example.com/search</span></span></code></pre></figure> <div tabindex="-1" class="heading-wrapper level-h4"><h4 id="restrict-who-can-query-your-content">Restrict who can query your content</h4><a class="anchor-link" href="#restrict-who-can-query-your-content"><span aria-hidden="true" class="anchor-icon"><svg width="16" height="16" viewBox="0 0 24 24"><path fill="currentcolor" d="m12.11 15.39-3.88 3.88a2.52 2.52 0 0 1-3.5 0 2.47 2.47 0 0 1 0-3.5l3.88-3.88a1 1 0 0 0-1.42-1.42l-3.88 3.89a4.48 4.48 0 0 0 6.33 6.33l3.89-3.88a1 1 0 1 0-1.42-1.42Zm8.58-12.08a4.49 4.49 0 0 0-6.33 0l-3.89 3.88a1 1 0 0 0 1.42 1.42l3.88-3.88a2.52 2.52 0 0 1 3.5 0 2.47 2.47 0 0 1 0 3.5l-3.88 3.88a1 1 0 1 0 1.42 1.42l3.88-3.89a4.49 4.49 0 0 0 0-6.33ZM8.83 15.17a1 1 0 0 0 1.1.22 1 1 0 0 0 .32-.22l4.92-4.92a1 1 0 0 0-1.42-1.42l-4.92 4.92a1 1 0 0 0 0 1.42Z"></path></svg></span></a></div> <p>Once your endpoint is on your own domain, you can put <a href="https://developers.cloudflare.com/ai-search/configuration/retrieval/public-endpoint/cloudflare-access/">Cloudflare Access</a> in front of it. For example, you usually want to give <code>/mcp</code> to specific agents rather than to anyone who finds the URL. Agents authenticate with an Access service token, and people who open the endpoint in a browser sign in through your identity provider.</p> <div tabindex="-1" class="heading-wrapper level-h4"><h4 id="search-several-instances-from-one-url">Search several instances from one URL</h4><a class="anchor-link" href="#search-several-instances-from-one-url"><span aria-hidden="true" class="anchor-icon"><svg width="16" height="16" viewBox="0 0 24 24"><path fill="currentcolor" d="m12.11 15.39-3.88 3.88a2.52 2.52 0 0 1-3.5 0 2.47 2.47 0 0 1 0-3.5l3.88-3.88a1 1 0 0 0-1.42-1.42l-3.88 3.89a4.48 4.48 0 0 0 6.33 6.33l3.89-3.88a1 1 0 1 0-1.42-1.42Zm8.58-12.08a4.49 4.49 0 0 0-6.33 0l-3.89 3.88a1 1 0 0 0 1.42 1.42l3.88-3.88a2.52 2.52 0 0 1 3.5 0 2.47 2.47 0 0 1 0 3.5l-3.88 3.88a1 1 0 1 0 1.42 1.42l3.88-3.89a4.49 4.49 0 0 0 0-6.33ZM8.83 15.17a1 1 0 0 0 1.1.22 1 1 0 0 0 .32-.22l4.92-4.92a1 1 0 0 0-1.42-1.42l-4.92 4.92a1 1 0 0 0 0 1.42Z"></path></svg></span></a></div> <p>A namespace can expose its own <a href="https://developers.cloudflare.com/ai-search/configuration/retrieval/public-endpoint/namespace/">public endpoint</a> with <code>/search</code>, <code>/chat/completions</code>, and <code>/mcp</code> paths that fan out across the instances you choose:</p> <figure class="nb-code-figure" data-nb-lang="bash"><pre class="astro-code astro-code-themes github-light github-dark nb-shiki-c6xiwz" tabindex="0" data-language="bash" data-nb-lang="bash"><code><span class="line"><span class="nb-shiki-1t8gfj">curl</span><span class="nb-shiki-mdbnqw"> https://ns-</span><span class="nb-shiki-1itgoe">&lt;</span><span class="nb-shiki-mdbnqw">NAMESPACE_ENDPOINT_I</span><span class="nb-shiki-140thh">D</span><span class="nb-shiki-1itgoe">&gt;</span><span class="nb-shiki-mdbnqw">.search.ai.cloudflare.com/search</span><span class="nb-shiki-dzsirb"> \</span></span> <span class="line"><span class="nb-shiki-dzsirb"> --header</span><span class="nb-shiki-mdbnqw"> "Content-Type: application/json"</span><span class="nb-shiki-dzsirb"> \</span></span> <span class="line"><span class="nb-shiki-dzsirb"> --data</span><span class="nb-shiki-mdbnqw"> '{</span></span> <span class="line"><span class="nb-shiki-mdbnqw"> "messages": [{ "content": "How do I configure AI Search?", "role": "user" }],</span></span> <span class="line"><span class="nb-shiki-mdbnqw"> "ai_search_options": { "instance_ids": ["docs", "support"] }</span></span> <span class="line"><span class="nb-shiki-mdbnqw"> }'</span></span></code></pre></figure> <div tabindex="-1" class="heading-wrapper level-h4"><h4 id="index-your-sites-without-a-sitemap">Index your sites without a sitemap</h4><a class="anchor-link" href="#index-your-sites-without-a-sitemap"><span aria-hidden="true" class="anchor-icon"><svg width="16" height="16" viewBox="0 0 24 24"><path fill="currentcolor" d="m12.11 15.39-3.88 3.88a2.52 2.52 0 0 1-3.5 0 2.47 2.47 0 0 1 0-3.5l3.88-3.88a1 1 0 0 0-1.42-1.42l-3.88 3.89a4.48 4.48 0 0 0 6.33 6.33l3.89-3.88a1 1 0 1 0-1.42-1.42Zm8.58-12.08a4.49 4.49 0 0 0-6.33 0l-3.89 3.88a1 1 0 0 0 1.42 1.42l3.88-3.88a2.52 2.52 0 0 1 3.5 0 2.47 2.47 0 0 1 0 3.5l-3.88 3.88a1 1 0 1 0 1.42 1.42l3.88-3.89a4.49 4.49 0 0 0 0-6.33ZM8.83 15.17a1 1 0 0 0 1.1.22 1 1 0 0 0 .32-.22l4.92-4.92a1 1 0 0 0-1.42-1.42l-4.92 4.92a1 1 0 0 0 0 1.42Z"></path></svg></span></a></div> <p>Website data sources support a new <code>discover</code> <a href="https://developers.cloudflare.com/ai-search/configuration/data-source/website/parse-types/">parse type</a>. It starts at the source URL and collects pages from both your sitemaps and the links it finds while crawling:</p> <figure class="nb-code-figure" data-nb-lang="bash"><pre class="astro-code astro-code-themes github-light github-dark nb-shiki-c6xiwz" tabindex="0" data-language="bash" data-nb-lang="bash"><code><span class="line"><span class="nb-shiki-1t8gfj">curl</span><span class="nb-shiki-dzsirb"> -X</span><span class="nb-shiki-mdbnqw"> POST</span><span class="nb-shiki-mdbnqw"> "https://api.cloudflare.com/client/v4/accounts/&lt;ACCOUNT_ID&gt;/ai-search/instances"</span><span class="nb-shiki-dzsirb"> \</span></span> <span class="line"><span class="nb-shiki-dzsirb"> -H</span><span class="nb-shiki-mdbnqw"> "Authorization: Bearer &lt;API_TOKEN&gt;"</span><span class="nb-shiki-dzsirb"> \</span></span> <span class="line"><span class="nb-shiki-dzsirb"> -H</span><span class="nb-shiki-mdbnqw"> "Content-Type: application/json"</span><span class="nb-shiki-dzsirb"> \</span></span> <span class="line"><span class="nb-shiki-dzsirb"> -d</span><span class="nb-shiki-mdbnqw"> '{</span></span> <span class="line"><span class="nb-shiki-mdbnqw"> "id": "my-ai-search",</span></span> <span class="line"><span class="nb-shiki-mdbnqw"> "type": "web-crawler",</span></span> <span class="line"><span class="nb-shiki-mdbnqw"> "source": "example.com",</span></span> <span class="line"><span class="nb-shiki-mdbnqw"> "source_params": {</span></span> <span class="line"><span class="nb-shiki-mdbnqw"> "web_crawler": {</span></span> <span class="line"><span class="nb-shiki-mdbnqw"> "parse_type": "discover",</span></span> <span class="line"><span class="nb-shiki-mdbnqw"> "discover_options": { "source": "links", "limit": 5000, "depth": 3 }</span></span> <span class="line"><span class="nb-shiki-mdbnqw"> }</span></span> <span class="line"><span class="nb-shiki-mdbnqw"> }</span></span> <span class="line"><span class="nb-shiki-mdbnqw"> }'</span></span></code></pre></figure> <p>To learn more, refer to the <a href="https://developers.cloudflare.com/ai-search/">AI Search documentation</a>.</p>