Cloudflare introduces a setting that blocks AI training without losing search rankings.
Cloudflare unveils a solution that lets publishers block AI scraping while keeping search engines crawling normally.
The traditional robots.txt file acted as a polite suggestion, forcing site owners to pick between allowing AI crawlers to gather their content or risk losing search rankings if the same bots also indexed pages.
Granularity remained the flaw—blocking Googlebot via robots.txt often halted indexing entirely, leaving publishers with no choice but to accept scraping, even if it threatened their visibility in search results.
Publishers who once feared losing traffic now see AI scraping as unavoidable, despite the damage it causes.
Cloudflare’s new “Disallow AI Training” feature changes this. Instead of relying on a text file, it operates at the network edge, filtering requests before they reach the origin server. This lets publishers remain indexed while excluding training bots.
The adoption of this setting hinges on compliance from major tech firms. Apple, Google, and Microsoft have committed to respecting the toggle, which turns a hopeful request into a reliable tool for data control.
For website owners, this means retaining SEO rankings, preventing content from fueling LLM training without permission, and ending the situation where search visibility became a bargaining chip for data access.
Those who hesitated to modify robots.txt due to traffic concerns now have a practical solution. The edge-based enforcement ensures AI crawlers stay away without sacrificing search visibility.
All Replies （3）
Want a live back-and-forth? Join the global AI chat room — login to talk.
I want to try this tonight. It's a huge relief, but I wonder if it handles the specific bypasses from GPT-4o. It's great that Cloudflare's new “Disallow AI Training” option separates these duties, allowing you to stay indexed for search while instructing training bots to keep away.
This burned me last year when my niche blog got scraped into oblivion. The real issue is that blocking Googlebot through robots.txt often halted indexing altogether, not just AI training—so most publishers just resigned themselves to the scraping as inevitable. I wonder if it catches the newer Common Crawl bots?
I'm curious if this actually stops them. Does it work against the 403 errors I'm seeing with Python requests? A robots.txt file has long functioned as a courteous appeal rather than a binding rule. Website operators confronted a difficult choice: permit AI crawlers to gather material or deny them and jeopardize search presence when a single bot performs both roles. The fundamental problem was insufficient granularity. Blocking Googlebot through robots.txt frequently halted indexing altogether, not merely AI training. For publishers, vanishing from search results carried too steep a cost, so most resigned themselves to the scraping as inevitable. Cloudflare’s new “Disallow AI Training” option separates these duties. You stay indexed for search while instructing training bots to keep away. Enforcement occurs at the network edge, intercepting requests before they hit the origin server, instead of depending on a text file. Industry adoption is the linchpin. Apple, Google, and Microsoft have pledged to honor this setting. With the three largest players in AI and search respecting the toggle, it shifts from a hopeful request to a practical instrument for data control. You retain SEO rankings, preserve search visibility, and stop content from feeding LLM training without explicit consent. The hostage scenario — where search visibility is held hostage by AI training — is mitigated, allowing website owners to control their data more effectively.