Browser Run - Crawl results include URL outcomes and robots.txt diagnostics

<p>Browser Run <a href="https://developers.cloudflare.com/browser-run/quick-actions/crawl-endpoint/#request-results-of-the-crawl-job"><code>/crawl</code> results</a> now include more detailed diagnostics. Each URL record can include an <code>outcome</code> that identifies the specific processing result, such as <code>success</code>, <code>robots_disallowed</code>, <code>timeout</code>, or <code>page_error</code>.</p> <p>The job-level <code>robots</code> array reports the <code>robots.txt</code> state for up to 50 origins encountered by the crawl. Each entry includes the origin, whether usable rules were available, and the HTTP status when the crawler received a response.</p> <figure class="nb-code-figure" data-nb-lang="json"><pre class="astro-code astro-code-themes github-light github-dark nb-shiki-c6xiwz" tabindex="0" data-language="json" data-nb-lang="json"><code><span class="line"><span class="nb-shiki-140thh">{</span></span> <span class="line"><span class="nb-shiki-dzsirb"> "robots"</span><span class="nb-shiki-140thh">: [</span></span> <span class="line"><span class="nb-shiki-140thh"> {</span></span> <span class="line"><span class="nb-shiki-dzsirb"> "origin"</span><span class="nb-shiki-140thh">: </span><span class="nb-shiki-mdbnqw">"https://example.com"</span><span class="nb-shiki-140thh">,</span></span> <span class="line"><span class="nb-shiki-dzsirb"> "state"</span><span class="nb-shiki-140thh">: </span><span class="nb-shiki-mdbnqw">"available"</span><span class="nb-shiki-140thh">,</span></span> <span class="line"><span class="nb-shiki-dzsirb"> "status"</span><span class="nb-shiki-140thh">: </span><span class="nb-shiki-dzsirb">200</span></span> <span class="line"><span class="nb-shiki-140thh"> }</span></span> <span class="line"><span class="nb-shiki-140thh"> ],</span></span> <span class="line"><span class="nb-shiki-dzsirb"> "records"</span><span class="nb-shiki-140thh">: [</span></span> <span class="line"><span class="nb-shiki-140thh"> {</span></span> <span class="line"><span class="nb-shiki-dzsirb"> "url"</span><span class="nb-shiki-140thh">: </span><span class="nb-shiki-mdbnqw">"https://example.com/docs/"</span><span class="nb-shiki-140thh">,</span></span> <span class="line"><span class="nb-shiki-dzsirb"> "status"</span><span class="nb-shiki-140thh">: </span><span class="nb-shiki-mdbnqw">"completed"</span><span class="nb-shiki-140thh">,</span></span> <span class="line"><span class="nb-shiki-dzsirb"> "outcome"</span><span class="nb-shiki-140thh">: </span><span class="nb-shiki-mdbnqw">"success"</span></span> <span class="line"><span class="nb-shiki-140thh"> }</span></span> <span class="line"><span class="nb-shiki-140thh"> ]</span></span> <span class="line"><span class="nb-shiki-140thh">}</span></span></code></pre></figure> <p>For field definitions and all possible outcome values, refer to <a href="https://developers.cloudflare.com/browser-run/quick-actions/crawl-endpoint/#url-outcomes">URL outcomes</a> and <a href="https://developers.cloudflare.com/browser-run/quick-actions/crawl-endpoint/#robotstxt-diagnostics">robots.txt diagnostics</a>.</p>