Can Cloudflare's AI pay per crawl succeed? I doubt it.
Cloudflare made the rounds earlier in the month with their announcement of their pay per crawl service. The idea is to enable content publishers to charge AI bots for access to their sites.
I’ve been involved in many facets of content for a long time, and I’m struggling to see how it reaches product-market fit.
Neither Cloudflare, nor any other service, will ever be able to block all scrapers. They can make their operations more expensive, at best; but not by much, since companies across the world are racing to build smarter, cheaper crawling machines.
You charge too high a price for your content, people will take the scraping route. What’s a low enough price point that would work? My guess is somewhere around $0.10 per article.
Let’s now flip to the publisher’s perspective. Are you okay with earning just 10 cents from the redistribution of your work to millions? Unlikely.
Thanks to the sea of SEO blog-spam (proud contributor here!) LLMs have become a convenient way to search. And you do want them to talk about you, because otherwise those users will be sent to your competitors.
