Cloudflare finally lets you block AI training without nuking your search rankings

Stop pretending that a robots.txt file is a magic shield. We all know it's basically a "please don't" sign that half the internet ignores. For the longest time, we were stuck in this hostage situation: either let the big tech giants scrape your soul for their next LLM, or block their crawlers and vanish from search results because they use the same bot for both.

Cloudflare just dropped a "Disallow AI Training" setting that actually separates these two. Now you can stay indexed for search while telling the training bots to beat it. Apparently, Apple, Google, and Microsoft are actually playing ball and have committed to honoring this setting.

Why is this a big deal?

Because the math of the internet is broken. Less than 1% of Cloudflare sites block search bots—obviously, we all want the traffic—but 17% of sites are actively trying to block AI training. The problem is "mixed-use crawlers." If you block the bot to save your data from a training set, you accidentally delete your existence from the search index.

Cloudflare is trying to fix this at the network level. Instead of hoping a bot reads your text file, the network identifies who is crawling and why, then blocks the ones that decide to ignore your preferences. They're tracking this on Radar so you can actually see who is lying to you.

What does "Accountable" actually mean?

Cloudflare created this "Accountable" label for bot operators who aren't just making vague promises. To get this badge, a company has to meet or commit to these specific requirements:

• Opt-out of training: Provide a way to opt out via robots.txt or similar standards.
• Opt-out of summaries: Give a way to opt out of AI summaries (currently done directly with the operator, but Cloudflare says they'll let you do this through their dashboard by early next year).
• URL visibility: Show exactly which pages were used for training and provide metrics on search appearances.
• No retaliation: Guarantee that opting out of training won't tank your traditional search rankings.

Google, Microsoft, and Apple all fit this description, though some of their "commitments" are time-bound rather than already implemented.

How the new security settings work

Cloudflare is now classifying bots by their specific behavior rather than just their name. You can now toggle controls based on three distinct categories:

• Search: Bots crawling to build an index.
• Training: Bots crawling to fine-tune or train a model.
• Agent: User-directed bots, like chat fetchers or browser-use agents, visiting on behalf of a human.

It's about time we got granular controls. A site-wide "yes/no" for AI is too blunt, especially when it comes to AI summaries. How much of your content gets scraped into a summary determines if a human ever actually clicks through to your site. If the summary is too good, you're just providing free training data while your traffic dies.