SAN FRANCISCO: OpenAI, Google, Microsoft and Anthropic pledged on Friday to give media outlets and other websites a single way to prevent their content from being scraped to train artificial intelligence (AI) models, amid growing criticism and legal battles involving news publishers.
Publishers currently have to implement multiple blocking commands for every bot that crawls the web to collect data. However, the proposed solution will allow them to issue a single, standardised instruction that is recognised and honoured by any AI company adopting the protocol.
The solution will take several months to be fully implemented.
Microsoft will begin implementing it on Feb 1, 2027, while Google plans to do so sometime in 2027. OpenAI, the maker of ChatGPT, said it would roll out the feature “in the coming months”, while Anthropic, the maker of Claude, said it had “already started” doing so.
The announcement comes as publishers face a surge in bots scraping their articles to feed AI models. Some, including The New York Times, have taken the matter to court.
In December 2023, The New York Times sued OpenAI and Microsoft, alleging that they had used millions of its articles without authorisation or payment to train ChatGPT. The two technology companies have invoked an exception to copyright law known as fair use.
Meanwhile, internet infrastructure provider Cloudflare has blocked AI bots by default on new customer websites since July 2025, while a standard known as Really Simple Licensing already allows publishers to charge bots for visiting their pages.
The new setting will apply only to future model training and will not prevent bots from crawling websites for other purposes. Content scraping will also remain permitted by default until a publisher changes the setting, Microsoft said in a blog post.
However, opting out of AI training could reduce the likelihood of a publisher’s content appearing in AI applications and services, Microsoft and Google said in separate blog posts.
“We believe creating a common signal that conveys such preferences to multiple providers at once is an important step. And we look forward to working with other AI companies on the standards and their implementation across the industry,” Meta spokesperson Andy Stone wrote on X, expressing support for the initiative.
