The major internet Content Delivery Network (CDN), Cloudflare, has declared a significant shift in its operations by blocking AI web crawlers from accessing websites without explicit permission or compensation. This change, effective from July 1, aims to address the growing issue of AI crawlers slowing down websites, a problem reported by numerous website owners.
AI crawlers, such as OpenAI’s GPTBot and Anthropic’s ClaudeBot, have been known to generate massive volumes of automated requests, causing websites to become sluggish. For instance, GoogleBot reports that cloud-hosting service Vercel bombards its hosted sites with over 4.5 billion requests monthly. These bots often crawl sites more aggressively than traditional search engine crawlers, revisiting pages frequently and sometimes hitting sites with hundreds of requests per second.
Cloudflare’s Game-Changing Move
In response to these challenges, Cloudflare now blocks AI crawlers by default for its two million-plus customers, representing 20% of the web. New websites signing up for Cloudflare’s services will automatically have AI crawlers blocked unless the site owner grants explicit permission. This policy change reverses the previous status quo, where website owners had to opt out of AI crawling.
The move is not only a response to frustrated website owners but also to publishing companies like The Associated Press and Condé Nast, who have expressed concerns over AI companies “strip mining” the web for content without compensation or consent. Often, these companies ignore standard protocols like robots.txt, which are meant to block crawlers.
“Numerous publishing companies are frustrated that AI companies have been ‘strip mining’ the web for content without compensation or consent.”
Legal and Economic Implications
Recent court cases have ruled in favor of AI companies like Meta and Anthropic, finding their use of copyrighted works legal under the doctrine of fair use. This has led to concerns among writers, artists, and publishers about the potential for AI to exploit their content without proper compensation. AI powerhouses such as OpenAI and Google continue to lobby for the classification of AI training on copyrighted data as fair use.
Cloudflare’s CEO, Matthew Prince, stated that the new policy is designed to “give publishers the control they deserve and build a new economic model that works for everyone—creators, consumers, tomorrow’s AI founders, and the future of the web itself.”
Pay Per Crawl: A New Economic Model
To complement the move to block AI crawlers, Cloudflare has introduced a “Pay Per Crawl” program. This initiative allows publishers to set their own rates for AI companies that wish to scrape their content. Currently in private beta, the program aims to create a framework where AI firms can pay for access or be denied if they refuse. This will be technically implemented using an old web server response, HTTP 402, which responds with a “Payment Required” error message.
“Until now, AI companies have not needed to pay for content licenses because they could simply take it without repercussions. Now they will need to negotiate.” — Nicholas Thompson, CEO of The Atlantic
Industry Reactions and Future Prospects
Cloudflare’s new policy has sparked reactions across the tech industry. AI companies have traditionally been against paying for content, with some arguing that requiring permission could hinder the AI industry’s growth. Sir Nick Clegg, former UK Deputy Prime Minister and Meta executive, warned that asking artists’ permission before scraping copyrighted content could “basically kill the AI industry.”
This policy is a direct response to the increasing volume and intrusiveness of AI crawlers and aims to stop the siphoning of traffic from publishers. Since the rise of AI, traffic to news sites has significantly decreased. For example, Business Insider’s traffic dropped by over 55% from April 2022 to April 2025. Without intervention, some predict that traffic from Google to news sites could drop to zero.
As the industry watches Cloudflare’s bold move, questions arise about whether other CDNs, such as Akamai, will follow suit. For now, the era of unrestricted AI crawling appears to be ending, at least for the portion of the internet that relies on Cloudflare’s services.
Stay informed with the latest developments by subscribing to our Tech Today newsletter, delivering the morning’s top stories straight to your inbox.
