Cloudflare says it has blocked 416 billion requests from AI bots attempting to scrape customer websites since July 1st. According to co-founder and CEO Matthew Prince, these protections are already reshaping how online content is accessed — and could fundamentally shift the economics of the open web.

The move builds on Content Independence Day, an initiative Cloudflare launched this summer alongside major publishers and AI companies. Its core principle is simple: AI crawlers should be blocked by default unless developers have a paid agreement granting access.

Since mid-2024, Cloudflare has offered customers built-in tools that automatically filter traffic from bots gathering data for AI training. The result: over 416 billion blocked crawl attempts in just a few months.

Prince argues that for decades the internet followed a predictable economic model: creators publish content, attract traffic, and monetize through ads, subscriptions, or sales. Generative AI disrupts this model. Large models now collect and reuse content without sending users back to the original publishers, instead serving instant answers through chat interfaces. That means creators lose traffic — and revenue.

Cloudflare’s new strategy, Prince says, aims to prevent a future where the web is reduced to just a few dominant AI platforms. While the company has historically focused on performance and security, the surge of generative AI has pushed it to defend the open internet as a place where independent creators, small media outlets, and niche communities can still thrive.

Prince also criticizes Google’s approach to web crawling. In its race to lead in AI, Google merged its search indexing bot with its AI training bot. The consequence: if a site blocks Google’s AI crawler, it disappears from Google Search. This forces publishers into a corner — they want to prevent unauthorized AI training, but they cannot afford to lose the search traffic that keeps them alive.

“You can’t use your monopoly in yesterday’s market to dominate tomorrow’s,” Prince says, calling Google the biggest obstacle to a healthy internet ecosystem. He argues Google must separate its search and AI crawlers so websites can control access independently.

Cloudflare also released new data showing the scale of Google’s advantage. Google currently sees:

3.2× more pages than OpenAI
4.6× more than Microsoft
4.8× more than Anthropic or Meta

This gives Google an unparalleled view of the web — and an enormous advantage in both search and AI.

Despite the power imbalance, Prince says early results from blocking AI crawlers are encouraging. Human-generated content, from local journalism to Reddit conversations, remains extremely valuable training material for AI systems. This opens the door for creators to negotiate paid licensing deals and specialized partnerships with AI developers, instead of giving their data away for free.

Prince acknowledges that long-term solutions may require legislation, but for now Cloudflare is using its infrastructure and market influence to push for a more pluralistic AI ecosystem — one where economic value is shared rather than concentrated.

He compares the situation to a superhero franchise: the hero of the last film becomes the villain of the next. In this story, he says, Google has become the antagonist, slowing the internet’s evolution. And until Google separates its search engine from its AI crawler, it will remain nearly impossible for creators to fully control how their content is used.

Source