News organisations don’t have a “genuine choice” about whether to block Google from scraping their content for its AI services, one publisher ha

News organisations are forced to accept Google AI crawlers, says FT policy chief

submited by

Style Pass

2024-11-06 18:00:16

News organisations don’t have a “genuine choice” about whether to block Google from scraping their content for its AI services, one publisher has warned.

Matt Rogerson, director of global public policy and platform strategy at the FT and former Guardian Media Group director of public policy, argued that Google’s “social contract” with publishers – through which it provided value to the industry by sending traffic to their sites – has been broken.

This is because Google now publishes summaries of those publishers’ articles in AI Overviews at the top of many search results – and also sells data generated via its search crawler to third-party large language models (LLMs).

In September last year Google introduced Google-Extended, a control which allows website owners to block its AI chatbot Gemini (formerly Bard) and its AI development platform Vertex from scraping their content.

However Google-Extended does not stop sites from being accessed and used in Google’s AI Overviews summaries, meaning that to avoid this publishers would have to opt out of being scraped by Googlebot, which indexes for search.