On September 15, 2026, Cloudflare changed the default rules for which AI systems are allowed to crawl the websites it sits in front of. If your site loads through Cloudflare — and a large share of professionally built sites do, without ever advertising it — this is a change to your site, whether or not anyone told you about it. The good news: it's a setting you control, not a rule imposed on you. The catch: the new default may not match what your business actually wants. Here's what changed, whether it affects you, and the short check worth running this week.
What actually changed
Cloudflare replaced a single, blunt "block AI bots" toggle with three separate categories:
- Search — crawlers that feed search engines like Google, so your pages can appear in ordinary search results.
- Agent — AI assistants fetching one of your pages on a person's behalf, like a chatbot pulling up your hours or your services when someone asks.
- Training — crawlers that harvest your content in bulk to train AI models, whether or not a specific person ever asked for your page.
Splitting the traffic this way is sensible. Those three activities are genuinely different, and treating "a model scraping your whole site to train on" the same as "a customer's assistant looking up your phone number" never made much sense. The part worth paying attention to is the default that comes with the split.
Who the new default affects — and who it doesn't
This isn't a switch that silently flipped for everyone. The new blocking behavior lands unevenly:
- On new sites and existing free-tier accounts, Agent and Training crawlers are now blocked by default on pages that run ads.
- Paid zones keep whatever settings they already had. Cloudflare didn't reach in and change configurations that were already set deliberately.
Even on a paid plan that kept its settings, though, the meaning of those settings shifted underneath them — "block AI bots" now maps to the new three-category model. So it's worth confirming that what you had configured still does what you think it does.
No, your site isn't about to vanish from Google
Some coverage of this change has framed it as websites being about to "disappear from Google." That isn't what it does, and it's worth treating that framing as clickbait until someone actually demonstrates the mechanism. Search crawlers are their own category specifically so that ordinary search visibility isn't collateral damage. The honest version is narrower and more useful: a default changed, it affects some accounts and not others, and the only way to know which bucket you're in is to look.
Should you block AI crawlers, or allow them?
This is the question the headlines skip, because the answer is "it depends on your business" — which doesn't make for a scary title.
If your website exists to get found and generate leads — which describes most small and mid-sized service businesses — you generally want to be readable. When someone asks an AI assistant "who does lawn care near me" or "who builds websites for contractors," you want your pages to be eligible to be part of that answer. Blocking the Agent category by default can quietly work against you.
If your website's value is its content — a publisher, a paid research library, a business whose articles are the product — then blocking Training crawlers is often exactly right. You don't want your work scraped in bulk to train a model that then answers questions without ever sending a visitor back to you.
Most small businesses are firmly in the first group. The takeaway isn't "AI good" or "AI bad." It's that the right setting for a lead-generation site is frequently the opposite of the right setting for a content publisher — and the new default leans toward the publisher's answer. Leaving it unexamined means accepting a default that wasn't chosen with your goals in mind.
The five-minute check to run this week
If Cloudflare sits in front of your site, this is worth ten minutes, not a project:
- Confirm you're actually on Cloudflare. If you're not sure, that's the first thing to establish — plenty of sites run through it without the owner knowing.
- Find the AI Crawl Control / bot-management settings in the Cloudflare dashboard for your zone. That's where the Search, Agent, and Training categories live.
- Decide, on purpose, what each category should do using the lead-gen vs. publisher distinction above. For most service businesses, that means allowing Search and Agent so AI-driven lookups can reach you, and making a deliberate choice about Training.
- Save it, and write down what you chose so the next person who looks at the account knows it was intentional, not an accident of the default.
That's the whole job. It isn't urgent in the "your site is down" sense — nothing broke on September 15 — but it's the kind of setting that quietly shapes who can find you, and it's easy to leave on autopilot for a year without realizing a default made the decision for you.
Where this fits if we host your site
Edge configuration — DNS, CDN, and security settings at the Cloudflare layer — is part of our standard Website Hosting, not an add-on. Changes like this one are exactly why that layer is managed rather than left to whatever the default happened to be. When a provider shifts a default that affects who can reach client sites, checking it across the sites we host is our job — not a homework assignment we hand back to you.
If your site runs on Cloudflare and you're not sure what its current AI-crawler settings are — or whether this change even applies to your account — that's a five-minute check we're happy to run for you. Get in touch and we'll take a look.