Cloudflare to Block AI Agent Crawlers by Default Starting September 15
## AI Agent Crawlers Now Require Permission: Here’s How to Obtain It
AI agent crawlers, the automated bots that retrieve online information for users seeking answers, will be blocked by default on certain web pages starting September 15. Cloudflare announced this alteration on July 1, shifting focus from just Google to all agents needing access.
Cloudflare has revamped its approach from a single "block-AI-bots" setting to a more nuanced framework with three classifications. The "Search" category pertains to bots that index pages for future references. The "Agent" category includes bots operating in real time on behalf of users, such as ChatGPT’s fetching features, while the "Training" category involves crawlers that gather content for artificial intelligence models. These revisions went into effect on July 1 for all customers, including those on the free tier.
As of September 15, the default settings will change. Pages displaying advertisements will block both the "Training" and "Agent" bot categories, although the "Search" category will remain active. This new default rule is applicable to newly onboarded Cloudflare domains, sites established by current customers, and all existing free-tier clients.
Any users who wish to maintain access for their bots must adjust their security settings before this cutoff date. Cloudflare maintains that the presence of advertisements indicates a page is designed for human visitors rather than automated requests. In this framework, a search crawler directing a user back constitutes a referral, while a bot retrieving and passing on information does not.
The new restrictions pose challenges for AI agent deployments that have relied on unrestricted access to the web. Applications that monitor competitor pricing or gather supplier announcements now face potential limitations. Until now, these operations did not require special licenses or permissions.
Cloudflare holds a significant portion of global web traffic, with its blocks functioning at the network level, unlike the traditional robots.txt directives that crawlers can often disregard. Ad-supported sites are precisely where agents operate as they house news, reviews, and pricing data. For enterprises relying on agents, the real risk lies not in legal action but in receiving no data at all.
A crucial complication arises with Google, which uses a single bot for both search and training purposes. As Cloudflare CEO Matthew Prince noted, the hope is that these changes will prompt mixed-use crawlers to differentiate between their search, agent, and training processes, emphasizing the importance of compliance.
For agents seeking permission to operate, the first step is identifying which of their Cloudflare accounts fall under the "Agent" classification. This classification hinges on the behavior of the application rather than an explicit opt-in process. Therefore, even real-time browsing agents will be categorized as crawlers regardless of the operator's intentions.
Expect coverage to lessen rather than cease entirely, as the blocks pertain to ad-supported pages while leaving others accessible. Negotiated access will become necessary rather than merely altering user-agent strings.
Publishers must first confirm their tier status, given that existing free-tier customers will automatically transition to the new defaults on September 15. They need to carefully evaluate whether blocking the "Training" category is worthwhile, since doing so also affects the ability of Googlebot to access their site, thus impacting search visibility.
The evolving payment structure for web crawlers reflects a shift towards more transactional relationships. Companies like Ceramic.ai are compensating publishers for content cited in AI search results, while You.com is paying agents that access premium content. Cloudflare also reports that over half of AI crawler traffic consists of re-fetching unchanged pages, highlighting inefficiencies on both ends.
This marks the inaugural phase in the evolving landscape of content access, shifting from free access to a more regulated approach. Agents who secure access prior to September will navigate the landscape effectively, while those who discover blockages later will face sudden adjustments.
