one year on
Cloudflare flips the default: AI crawlers now blocked on new sites
The company that fronts roughly a fifth of the web will block AI training bots by default, and pilots 'Pay Per Crawl' — asking whether the age of free scraping is ending.
Cloudflare announces that new websites on its network will block AI crawlers by default, reversing the web’s decades-old open-scraping norm. Site owners must now opt in to being training data.
Alongside the default flip, the company launches a private beta of Pay Per Crawl: a marketplace where publishers can set a price for bot access, with Cloudflare as the toll booth. CEO Matthew Prince frames it bluntly — the deal where search engines sent traffic in exchange for content is dead, because AI answers don’t send traffic.
The announcement lands with signed support from Condé Nast, TIME, The Atlantic, and dozens of other publishers.
Whether this is the start of a functioning content economy or an unenforceable gesture is this week’s argument. What nobody disputes: about 20% of the web just changed its posture toward AI overnight.
Major outlets line up in support within hours; press releases use the phrase 'content independence day.'
Point out that the labs most worth blocking already have the data, and that robots.txt was always a handshake, not a lock.
One year later — open only if you can handle spoilers
The scraping wars escalated all year — Cloudflare later accused AI companies of disguising crawlers, and content licensing deals became standard line items for the big labs. 'Who pays for the open web' turned out to be one of 2025's most durable stories.