Cloudflare’s Sept 15 AI Crawler Block Skips Most Sites

On 15 September 2026, Cloudflare changes what happens by default when an AI crawler arrives at a site it sits in front of. The coverage has mostly compressed this into “Cloudflare will block AI crawlers by default,” which is close enough to be useful and wrong in the one place that matters: the new default is keyed to whether a page displays ads. Most WordPress business sites do not display ads. On those sites, the 15 September change does nothing at all.

I run this site behind Cloudflare, so I went and measured what it would actually do here, and then measured the thing the announcement is really about — who is crawling in the first place. Over the 28 days to this morning, AI crawlers fetched more pages from this server than every search engine combined. Not slightly more. Between 1.26 and 1.73 times more, depending on how honestly you want to count.

What Cloudflare is actually changing

The announcement went up on Cloudflare’s blog on 1 July 2026, under the title Your site, your rules: new AI traffic options for all customers. It splits automated crawling into three categories rather than treating “bot” as one thing:

• Search — collecting and indexing your content so it can answer questions about it later, in a way that still sends people to you.
• Agent — software acting in real time on behalf of a person who wants something done right now.
• Training — crawling to train or fine-tune a model, where the content is absorbed into the model’s weights and there is no link back, ever.

From 15 September 2026, Cloudflare sets new defaults against those three categories. Training and Agent get blocked by default on pages that display ads. Search stays allowed. Cloudflare’s reasoning for using ads as the trigger is that an ad is evidence the page was built for a human being to land on and look at, which makes it a page with a business model that a training crawler quietly undercuts.

Two details from the announcement are worth pinning down, because the secondary coverage blurs both:

1. The scope is narrower than the headlines. Cloudflare describes the new defaults as applying to new domains onboarding to Cloudflare. If you have been a customer for three years, the announcement does not describe your existing configuration being rewritten underneath you. Site owners can also opt out of the new defaults in their security settings in advance.
2. Multi-purpose crawlers get the strictest rule that applies to them. A crawler that does search and training is treated as a training crawler when a site restricts training. That is the entire point of the September deadline: it is pressure on AI companies to split their crawlers by purpose and declare which is which, rather than running one bot that does everything and letting site owners guess.

The trigger is ads, and this site has none

Before treating this as news that affects me, I checked whether the trigger condition exists here. It does not, and it does not by a wide margin.

I queried every published post and page on this site for the markers of the ten largest ad networks — adsbygoogle, googlesyndication, doubleclick, data-ad-client, adsense, media.net, Taboola, Outbrain, Ezoic and Mediavine. Every one returned zero across all 648 published objects on the site — 402 posts, 67 pages and 179 other published records — with a control term (chatbot) returning 441 hits on the identical query, so the zeros are real rather than a broken query. Fetching the live homepage and grepping the rendered HTML for the same markers returned nothing either.

So on this site, the 15 September default is a no-op twice over: the domain is not newly onboarding, and there are no ad-bearing pages for the rule to attach to. I suspect that is true of most WordPress sites reading about this — SaaS sites, agency sites, local business sites, documentation sites and product blogs generally do not run display ads. The publishers this rule was designed for are a real and important constituency, but they are not the median WordPress install.

Which leaves the more interesting question. If the default is not going to decide this for me, what is actually arriving at the door?

Who actually crawls this site

I pulled the raw origin access log rather than a dashboard: the compressed August log plus the live one, covering 31 July 2026 08:14 to 28 August 2026 03:57. That is 331,206 requests. I extracted the user-agent field from every line and counted identified crawlers inside that field only.

The first thing that jumps out is that Amazonbot, at 10,171 requests, is the single most active identified crawler on this site — half again as busy as Googlebot at 6,738. Then ChatGPT-User (5,650), ClaudeBot (5,593) and Meta’s crawler (5,268) all land above Baiduspider and bingbot.

Two honest ways to add it up

Amazonbot is awkward to classify and I would rather show the awkwardness than resolve it quietly. Amazon documents it as serving Alexa and Amazon search as well as model training. Calling it an AI crawler inflates the AI total; calling it a search engine deflates it. So here is the count both ways.

Both readings land in the same place. Whichever denominator you prefer, AI crawlers pull more pages off this server than every search engine combined. That is the fact I would not have predicted before running the query, and it is the one that makes a robots.txt decision worth thinking about rather than deferring.

What they get, and the robots.txt that says nothing

Here is what the origin actually answered them with, plus how many times each crawler asked for robots.txt.

Three things in that table deserve a sentence each.

682 robots.txt fetches, for a file that names no AI crawler at all. This site’s robots.txt disallows some WooCommerce upload paths and /wp-admin/, allows admin-ajax.php, and points at the sitemap. There is no GPTBot directive, no ClaudeBot directive, no Google-Extended directive, no CCBot directive. OAI-SearchBot checked it 245 times and ClaudeBot 237 times over 28 days, and every one of those checks returned a policy that had nothing to say about them. The crawlers are asking. The site is not answering. That is a default nobody chose — it is what happens when a WordPress site is set up and the file is never revisited.

Google-Extended got 293 requests and zero successful responses. 145 were refused with a 403, 50 returned 404, and not one returned 200. It is the only row in the table like that. I can measure it and I am not going to invent a mechanism for it — Cloudflare sits in front of this origin and I did not change anything to produce it. Recorded as measured and unexplained.

The crawl is shallow and repetitive. Looking at what OpenAI’s, Anthropic’s and Perplexity’s crawlers actually fetched, the top paths are the homepage (1,470 requests), robots.txt (551), the sitemap index (396), an oEmbed endpoint (249), and then two documentation pages and the main plugin comparison. This is not a deep archive sweep. It is mostly repeated re-checking of the same handful of entry points.

The limitation that undercuts every number above

Everything in this post counts user-agent strings. A user-agent string is a claim the client makes about itself, and any client can make any claim.

I know exactly how much that matters here, because I measured it on this same site two weeks ago: when I verified self-declared crawlers by reverse DNS instead of trusting the header, 46% of the traffic claiming to be Googlebot was not Googlebot. The full method and results are in that write-up on verifying AI crawlers, and the honest consequence for today’s numbers is that they are an upper bound, not a census. Some share of the 37,111 is something else wearing an AI crawler’s name.

There is a second limitation running the other way. Cloudflare is in front of this origin, so the access log records what Cloudflare passed through. Anything Cloudflare challenged or blocked at the edge never reaches the origin and never appears in these counts. So the numbers understate total demand while overstating verified identity, and I would not want either caveat dropped when the figure gets quoted.

What to actually do about it on a WordPress site

If the September default will not reach you, the decision is yours to make explicitly. The useful move is to stop treating “AI crawlers” as one switch, because the three purposes have genuinely different consequences.

The two rows people most often get wrong are Google-Extended and the -User agents. Blocking Google-Extended is safe with respect to your Google Search rankings — it is a separate token that Googlebot does not honour. But blocking ChatGPT-User or Claude-User is not blocking a training crawl at all; it is refusing a fetch that a human being triggered by asking a question about your page. That is closer to blocking a referral than to protecting your content.

A middle position that costs nothing: allow the search and user-triggered agents, disallow the pure training crawlers, and write it down in robots.txt so the 682 checks a month get an actual answer.

Is AI visibility worth having in the first place?

This is where I would rather show my own numbers than repeat the industry line, because on this site they do not agree with it.

When I classified the queries this domain shows up for, the conversational, sentence-shaped ones — the fingerprint of a model grounding an answer rather than a person typing into a search box — accounted for roughly a quarter of page-one impressions and produced zero clicks across 2,760 impressions. The full breakdown is in the analysis of those page-one impressions with no clicks. Being visible to models has, so far, been worth measurably nothing here in traffic terms.

That is one domain and it is not a general law. But it is a reason to make this decision on evidence from your own logs and Search Console rather than on the assumption that AI visibility is obviously valuable. Pull the numbers before you pick a side.

The part your chatbot does not care about

Worth stating because it comes up: none of this touches an on-site chatbot’s knowledge base. A plugin like MxChat builds its knowledge from your own database and your own content, on your own server — it is not reading a third party’s crawl of you. You can disallow every AI crawler in the table above and your site’s own assistant answers exactly as well tomorrow as it does today. The two things get conflated constantly, and they are unrelated systems. The same goes for anything you have layered on top of it from the add-ons — none of them depend on a third party being allowed to crawl you.

What is related is the disclosure and data-handling side, which has real deadlines attached — see the notes on AI Act disclosure requirements for chatbots and on what your site retains and for how long.

FAQ

Does the 15 September change block AI crawlers on my existing site?
Based on Cloudflare’s own announcement, the new defaults are described for new domains onboarding to Cloudflare, and they attach to pages that display ads. If you are an existing customer without ads, expect nothing to change. Check your own settings rather than trusting that, including mine.

Will blocking Google-Extended hurt my Google rankings?
No. Google-Extended is a separate token covering Gemini training and grounding. Googlebot, which handles Search, does not obey it. You can disallow one and keep the other.

Do AI crawlers actually respect robots.txt?
The major named ones publish that they do, and on this site they demonstrably read the file — 682 fetches in 28 days. Whether a given request honours it is a separate question from whether the string in the log is genuine, which is the harder problem.

How do I see this on my own site?
Get your raw access log from your host rather than a plugin dashboard, extract the user-agent field, and count. A caching plugin’s stats will not show you crawler traffic reliably, and Search Console only shows you Google.

Is 11% of requests a lot?
It is more than every search engine put together on this domain, which is the comparison that made me stop and look. In absolute terms it is a small load. The reason to care is not bandwidth — it is that the largest identified category of automated visitor to this site is now something you have no relationship with, no report from, and no click coming back from.

The short version

• Cloudflare sets new AI crawler defaults on 15 September 2026, splitting crawling into Search, Agent and Training.
• Training and Agent get blocked by default only on pages that display ads, and the announcement scopes the new defaults to newly onboarding domains.
• This site has zero ad-network markers across all 648 published objects, so the default does not reach it. Most WordPress business sites are in the same position.
• Over 28 days and 331,206 requests, AI crawlers out-fetched every search engine combined — 37,111 vs 21,404 including Amazonbot, 26,940 vs 21,404 without it.
• Amazonbot alone (10,171) beats Googlebot (6,738).
• AI crawlers fetched robots.txt 682 times and got a file with no AI directives in it.
• Every count here is a user-agent count, and 46% of this site’s self-declared Googlebot was fake when verified. Treat the totals as an upper bound.