The Block Nobody Declared
Nine of 18 agency sites turned an AI crawler away. None of the nine said so in robots.txt. I tested 12 AI crawlers against 18 marketing agencies that sell search visibility on 16 September 2026, and every refusal came from the server or the CDN, the layer most audits never check.
A week earlier the count was one of nine. One site declared its block in robots.txt, and its server ignored the declaration. Eight days later that site had swapped sides: the file welcomed everyone and the server did the refusing. As far as I can tell, nobody at the agency touched a thing.
That swap is the reason this piece exists.
Two layers, and why almost everyone reads only one
The question I asked about each site is narrow on purpose. Not whether an assistant recommends it. Whether an AI crawler can fetch its home page at all.
The first layer is robots.txt, the public text file at the root of a domain that tells crawlers which paths they may request. The standard that defines it says plainly that its rules aren't a form of access authorization. It's a request, it's free to read, and it's where nearly every audit stops.
The second layer is the response. A server can hand your browser the page and refuse the same page to GPTBot with a 403, before any of the site's own code runs. Nothing about that shows up in robots.txt. It comes from a CDN setting, a security plugin or a host firewall.
So for each crawler I checked both: the robots.txt rule that applies to it, and a real request for the home page carrying its user agent. Two of the 12, Google-Extended and Applebot-Extended, have no user agent of their own, so for those two robots.txt is the only signal there is.
The count on 16 September 2026
I left out one of the 19 domains I track because it refused even a desktop browser. That leaves 18.
| What I measured | 8 September | 16 September |
|---|---|---|
| Agency sites measured | 18 | 18 |
| Turn away at least one AI crawler | 9 | 9 |
| Of those, declare it in robots.txt | 1 | 0 |
| Of those, only turn away Bytespider | 5 | 5 |
| Of those, turn away a crawler from OpenAI, Anthropic or Perplexity | 3 | 3 |
| Of the 18, bounce every bot, so the AI tests prove nothing | 1 | 1 |
| Publish a real llms.txt file | not comparable | 10 |
The honest reading starts with the fourth row. Five of the nine only refuse Bytespider, the crawler from ByteDance, and that block is common enough to be close to a default. My own site refuses it too. The ninth site refuses CCBot and Meta's crawler. The three that turn away crawlers from OpenAI, Anthropic or Perplexity are the ones that matter most for assistants, and none of them says so in its file.
The llms.txt figure isn't comparable because I fixed a flaw in my own script in between. On 8 September it counted a page that answered with HTML as if it were a real llms.txt file.
The case that changed in eight days
On 8 September, one agency's robots.txt opened with a block marked "BEGIN Cloudflare Managed content". It disallowed GPTBot, ClaudeBot, CCBot, Bytespider, Google-Extended, Applebot-Extended and meta-externalagent. Under it sat the site's own rule, which lets every crawler in. The server let them in too.
| Same site, home page | 8 September | 16 September |
|---|---|---|
| What robots.txt says | A Cloudflare block that disallows seven AI crawlers, then "allow everyone" | Only "allow everyone" and the sitemap |
| GPTBot, ClaudeBot, CCBot, Bytespider | 200 | 403 |
| OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, Perplexity-User | 200 | 200 |
| A made-up bot, as a control | 200 | 200 |
On 16 September the refusal was a 25-byte line of plain text, "Your request was blocked." A separate check with Amazon's crawler that night got the same answer. The search and assistant crawlers still got the full page.
Look at what flipped. On the 8th the file said no and the door said yes. On the 16th the file said yes and the door said no. Anyone who audited this site by reading robots.txt got the wrong answer both times.
Why a provider, and not the agency, probably did it
On 15 September, Cloudflare announced it was retiring two features: "Block AI Bots" and the managed robots.txt that wrote blocks like the one above into the file. AI traffic now runs through separate Search, Training and Agent controls per domain. Training got a new option, Disallow AI Training, which Cloudflare describes as keeping Googlebot, Bingbot and Applebot crawling for search while it blocks crawlers that only train, including those from Amazon, Anthropic, Meta and OpenAI. Cloudflare also says existing settings carry over and most customers don't need to do anything.
What I saw at the edge fits that option closely. Training crawlers got a 403, search and assistant crawlers got in. There's a second clue. The domain Cloudflare uses as the example in its managed robots.txt documentation had lost its managed block by 16 September too, and it refused GPTBot with the same 25-byte line.
This is an inference, and I'd rather label it than oversell it. From outside I can't see the site's settings, who changed what, or when. One piece doesn't fit, either. Disallow AI Training is supposed to publish a no-training line in robots.txt, and neither file had one on the 16th.
Still, it changes the story. When I first read this site, the point was that a vendor had written the block, not the owner. Now the vendor has rewritten it, moved it to a place the owner can't see from the file, and told customers they didn't need to act.
The case with no provider in sight
A second agency publishes the most permissive robots.txt there is, generated by an SEO plugin: one rule for every crawler, with an empty disallow. On both dates its server answered a desktop browser and my made-up bot with a 200, and answered seven named crawlers with a 403: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot and Perplexity-User.
This one is more serious than the first case. It shuts out the crawlers that fetch a page while an assistant answers a live question, not only the training ones. And its response headers point to a plain nginx server, not Cloudflare, so the provider story above doesn't cover it. The rule could live in a firewall, a plugin or a host preset. I can't tell which from outside.
Its three other crawlers, CCBot, Bytespider and meta-externalagent, got a 429 both times. I didn't count those as blocks.
What this does not prove
A 403 to a crawler I impersonated isn't proof that the real crawler gets a 403. My requests come from a home connection in Argentina, not from OpenAI's or Anthropic's addresses, and a CDN can treat a verified crawler differently. That cuts both ways, so I only report what the server said to the user agent I sent.
It doesn't prove anyone chose the block. In the first case the evidence points at a provider migration. In the second I can't see the cause at all.
It doesn't cover the whole market. The 18 sites are agencies I track, picked because they compete for the same kind of work I do. They aren't a random sample.
And being readable isn't being cited. An open door doesn't make a site the answer, the same way being indexed never made a page rank. Access is the first of the three questions on my method page. Whether a machine can quote you once it's inside is a different job, covered in making a site citable.
My own site, measured the same way
On 16 September, 11 of the 12 crawlers could read Contento. The twelfth was Bytespider, refused by my platform at the server, with nothing in my robots.txt. I didn't choose that. I know about it because I measure it, and that's the only difference I'd claim.
If you want the numbers for a whole niche rather than one site, I ran the same test on 54 therapy practice websites, and the method behind that index is written so you can repeat it.
How to check your own site
You don't need a tool or a vendor. A terminal, ten minutes and some patience will do. Six steps, and the ones people skip are the second, the third and the sixth.
First, read your robots.txt, including what's missing
Open yourdomain.com/robots.txt and read it top to bottom. Look for any block you didn't write, especially one wrapped in comments naming a vendor. If you're on Cloudflare and a managed block used to sit at the top, it may be gone since 15 September. A clean file doesn't mean crawlers get in. It means the file no longer tells you.
Second, get a baseline as a browser
Request your home page with a normal desktop browser user agent and note the status code and the size of the page. Every result after this is a comparison against it. Without a baseline, a 403 or a short page means nothing.
Third, run the control with a bot that has nothing to do with AI
Request the same page with a made-up bot name. If that gets a 403 too, your server refuses every bot, and the AI tests can't tell you anything until that changes. One site in my sample did exactly this, and I didn't count it among the nine.
Fourth, ask as each AI crawler, with the full user agent
Request the page once per crawler, a few seconds apart, with each crawler's full user agent string and not just its name. On the first site in this piece, the bare word "GPTBot" got the page and the full string got the 403. A 403 counts only when the browser and the made-up bot both got a 200. A 200 with a fraction of the browser's bytes is a soft block, so write that down too.
From a terminal, the GPTBot request looks like this:
curl -s -o /dev/null -w "%{http_code} %{size_download}" -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot" https://yourdomain.com/
Fifth, treat every 429 as your own fault
A 429 is a rate limit. It says how fast you asked, not what the site allows. Wait, retry once more slowly, and only record the result if it holds. Two agencies in my first run went from "blocks OpenAI and Anthropic" to open once I slowed down.
Sixth, if you're on Cloudflare, read the Training setting
Open the domain's security settings and find the Search, Training and Agent controls. Disallow AI Training is meant to keep you in Google, Bing and Apple search. Block, since 15 September, reaches Googlebot, Bingbot and Applebot as well. If you picked Block on purpose, confirm in Search Console or your server logs that Google still crawls you. Pretending to be Googlebot from your laptop won't answer that.
What to do with what you find
If you find a block you didn't put there, the fix is a setting, not a project. Usually it's in the CDN dashboard, a security plugin's bad bot list or the host's firewall.
Decide on purpose before you flip it. Refusing training crawlers while letting search and assistant crawlers in is a coherent position. Refusing everything is coherent too, if your content is the product. Paying for AI visibility while a setting you never read does the opposite isn't.
Questions I get about this
If my robots.txt allows everything, can AI crawlers read my site?
Not necessarily. On 16 September 2026, all nine agency sites in my sample that turned an AI crawler away had a robots.txt that allowed it. The file tells you what a site asks for. The server's response tells you what it does.
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot is OpenAI's training crawler. The crawlers that fetch pages for ChatGPT search and for users who ask it to open a link are different ones, with their own names. Blocking one doesn't block the others.
Did Cloudflare change how AI crawlers are blocked in September 2026?
Yes. On 15 September 2026 Cloudflare announced it was retiring Block AI Bots and its managed robots.txt, and added a Disallow AI Training option to its Training control. On one site I track, the robots.txt block disappeared and the server started refusing training crawlers in the same week.
Is a 403 to a crawler always a decision about AI?
No. Many sites refuse anything that isn't a browser. Unless a made-up bot unrelated to AI gets a 200 from the same page, a 403 to an AI crawler proves nothing about AI.
How often should I check?
Whenever your CDN, host or security plugin changes a plan or ships an update, and otherwise once a quarter. The block in the first case moved without anyone at the agency typing a rule, on the provider's schedule.
If you'd rather have someone read your server's answers with you, book a call and bring your domain.
