Cloudflare Threatens to Block Google Unless It Splits Search from AI Bots
The tech giants' favorite loophole of hiding greedy AI scrapers inside helpful search bots is officially under fire. Cloudflare is drawing a massive line in the sand, and Google is about to find out what happens when you try to eat your cake and crawl it too.
Starting September 15, 2026, the internet's favorite security guard, Cloudflare, is turning off the tap for any AI training bots and AI agents scraping ad-supported websites. This ban includes "hybrid" bots that try to play both sides, meaning Googlebot, Applebot, and BingBot face a total blackout unless their creators split their search indexing duties from their AI data-hoarding hobbies.
The technical reality is that Cloudflare has neatly sorted these digital pests into three buckets: Search, Agent, and Training. While website owners can toggle these on or off, the new default setting will automatically lock the gates on ad-heavy pages, under the very logical assumption that if a page has ads, it was made for humans, not for silicon-based freeloaders.
The main target of this digital hostage situation is Google, which has spent years hiding behind its search monopoly. Because website owners are terrified of disappearing from search results, they let Googlebot run wild, giving Google twice as much AI training data as its competitors. While Google offers an opt-out via its Google-Extended bot, Cloudflare has politely called this effort completely useless.
To back this up, Cloudflare dropped some brutal numbers showing that bots now make up 57.4% of all HTML traffic. The old social contract of "let us scan your site and we will send you visitors" is officially deceased. While Google scans a site 14 times for every single referral click, OpenAI takes 1,700 scans, and Anthropic needs a staggering 73,000 scans to send one lonely human. Worse, when Google shows its AI summaries, users click on actual links only 8% of the time.
In response, Cloudflare is pivoting its Pay Per Crawl program into a "Pay Per Use" model, where publishers get paid only when their content actually gets served in an AI response. To get the ball rolling, they have partnered with AI search engines You.com and Ceramic.ai, while giving webmasters a new dashboard called Attribution Business Insights to track exactly which bot is stealing their hard work.
The golden age of treating the open web as a free buffet for large language models is rapidly crashing into a paywall. If search giants refuse to separate their utility tools from their commercial AI vacuums, the web might just fracture into a series of highly guarded digital fortresses where search engines are no longer welcome.
Source: Cloudflare
Comments
This is where the magic happens: AI reads your discussion and rewrites the article based on the most interesting comments. Each strong comment adds points to the meter below. Once the meter is full, the article updates live — no page reload needed.