Cloudflare’s new marketplace will let websites charge AI bots for scraping
Cloudflare announced plans on Monday to launch a marketplace in the next year where website owners can sell AI model providers access to scrape their site’s content. The marketplace is the final step of Cloudflare CEO Matthew Prince’s larger plan to give publishers greater control over how and when AI bots scrape their websites.
“If you don’t compensate creators one way or another, then they stop creating, and that’s the bit which has to get solved,” said Prince in an interview with TechCrunch.
As the first step in its new plan, on Monday, Cloudflare launched free observability tools for customers, called AI Audit. Website owners will get a dashboard to view analytics on why, when, and how often AI models are crawling their sites for information. Cloudflare will also let customers block AI bots from their sites with the click of a button. Website owners can block all web scrapers using AI Audit, or let certain web scrapers through if they have deals or find their scraping beneficial.
A demo of AI Audit shared with TechCrunch showed how website owners can use the tool, which is able to see where each scraper that visits your site comes from, and offers selective windows to see how many times scrapers from OpenAI, Meta, Amazon, and other AI model providers are visiting your site.
