Does Cloudflare block AI crawlers?

RushilFounder, Ru Visibility2 October 2026

Short answer

It can, and for some sites it does by default. Cloudflare has settings that block AI crawlers, and its defaults for new domains have changed more than once since 2025. Check your own dashboard, under Security Settings, rather than assuming either way.

The settings

Cloudflare's documentation describes two places to manage this: “Security Settings → Block AI bots”, the older option, and “Security Settings → Configure AI bot policies”, the newer one, which groups bots into Search, Agent and Training (Cloudflare docs).

What the older setting blocks

Cloudflare says it “blocks verified bots that are classified as crawling for the purpose of AI training, as well as a number of unverified bots that behave similarly”, and that it “excludes mixed-purpose bots that are used both for Training and for Search.”

The defaults

On 1 July 2025 Cloudflare announced it was “changing the default to block AI crawlers unless they pay creators for their content” (Cloudflare blog). Its documentation now describes newer defaults: “On September 15, 2026, Cloudflare will set updated defaults for new domains: bots classified as Training or as Agent will be blocked on pages that display ads, and Search will remain allowed.”

Because the rules depend on when your domain was added and what someone has since switched on, the only reliable answer is to look.

How to check yours

Log in to Cloudflare and open your domain.
Go to Security Settings and find the AI bot options.
Make sure the crawlers that fetch pages for answers, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, are allowed if you want to appear in those tools.
If a developer or agency manages Cloudflare for you, ask them which AI bots are blocked and why.

Not using Cloudflare?

Other hosts, firewalls and WordPress security plugins have similar options. The same check applies: how to test whether AI crawlers can reach your site.

Sources

Find out what AI says about you.

Send us your web address. We run our own tool on it, free, and send you what it says. The whole thing — not a taster with the useful bits taken out.

Whether AI can even read your site

ChatGPT, Claude, Perplexity, Google and Bing all send a small program to read your pages. We knock on your door as each one and see who gets let in. If they are being turned away, nothing else matters until that is fixed.

Two scores out of 100 — one for Google, one for AI

Plus what is behind each number. Where we could only make a fair guess, we say so instead of dressing it up as a measurement.

A list of fixes, best one first

Ordered by what gives you the most for the least work, so the top of the list is the quickest win. It is yours either way — use it with us or without us.

We ran it on our own site too. It told us what we were getting wrong, and we published that. A tool you will not point at yourself is not worth much.

One thing the free check does not do: actually ask ChatGPT, Perplexity and Gemini for businesses like yours, dozens of times, and count how often your name comes up. That costs us real money each time we run it, so it comes with paid work.