Google, Microsoft, OpenAI and Anthropic Back One Shared AI Training Opt-Out for Publishers

Four of the world’s biggest artificial intelligence companies — Google, Microsoft, OpenAI and Anthropic — will honour a single shared signal that lets a website opt out of having its content used to train generative AI models. The move, announced on Friday, 30 October, answers a complaint publishers have pressed for years: keeping content out of training sets currently means tracking down each company’s crawler and writing a blocking rule for it by name.

FAKTA

• Google, Microsoft, OpenAI and Anthropic will honour a shared robots.txt signal, __usage_ai-training.
• A publisher adds one group to robots.txt; participating companies read it the same way.
• The token comes from an individual Internet-Draft by Anthropic’s Kevin Kelley, submitted to the IETF’s AI Preferences working group.
• Anthropic has already started support; Microsoft starts 1 February 2027; Google aims for early 2027; OpenAI aims to follow within months.
• The signal covers future training only: it does not remove already-collected content, stop crawling, or remove sites from search results.

AI training opt-out: how the shared signal works

Under the shared approach, a publisher adds a policy entry to the robots.txt file — the decades-old file that governs web crawlers — stating a preference about training use. The entry uses the token __usage_ai-training as a user-agent group, followed by Disallow lines marking what is off limits, with Allow lines able to make exceptions, for example for a press kit.

Until now, a publisher had to list GPTBot, Google-Extended, ClaudeBot and a growing pile of other crawlers one by one, then keep that list current as new crawlers appeared. The shared token is designed to replace that patchwork with one rule that every participating company reads identically.

Different timelines for the same pledge

The four companies have not promised the same start date. Anthropic said it has already started rolling out support. Microsoft will begin honouring the preferences from 1 February 2027 as its Bingbot crawler revisits pages, and will stop recognising its NoArchive and NoCache tags as training or grounding signals after 1 March 2027.

Google aims to support the shared control in 2027, with its crawler documentation narrowing that to early 2027; where a site opts out through the shared line without mentioning Google-Extended, Google plans to read it as if the publisher had disallowed Google-Extended. OpenAI said it aims to honour the signal within the next few months while its current robots.txt policy stays in effect.

What the signal does not do

The limits are as specific as the promise. The setting applies only to future training of generative models, and it does not prevent bots from crawling a site otherwise. Microsoft said its controls can distinguish AI training from AI grounding, in which a model is pointed to verified sources to improve answers, and that blocking grounding will significantly reduce a site’s visibility in AI-generated answers.

The draft is also not yet an official standard: it is an individual Internet-Draft that the IETF has not endorsed and that has no formal standing in the standards process. Microsoft stressed that the new preferences do not establish licensing or payment arrangements, and the reports note the signal says nothing about deleting existing datasets or paying publishers.

Conclusion

For publishers, the practical effect will develop gradually through 2027 as each company’s crawler adopts the shared rule. The direction, though, is clear: after years of fragmented opt-outs, the biggest AI companies have agreed that one line in robots.txt should be enough to say no to training.

Related Articles

• First AI-Designed Virus Given to a Patient in Medical Milestone
• DeepSeek and Moonshot Used OpenAI’s Astra Model to Train Latest AI Systems
• Malaysia Seizes $12.1 Million Worth of AI Chips at Kuala Lumpur Airport

Frequently Asked Questions

What is the new AI training opt-out?

A shared robots.txt signal, built around the token __usage_ai-training, that tells participating AI companies a site does not want its content used to train generative AI models.

Which companies support it?

Google, Microsoft, OpenAI and Anthropic, following announcements on 30 October 2026.

When does it take effect?

Anthropic has already started; Microsoft from 1 February 2027; Google in early 2027; OpenAI within the coming months.

Does it stop a site being crawled or appearing in search?

No. It covers training use only; it does not block crawling or remove sites from search results.

Does it delete content already collected for training?

No. It applies to future training only and says nothing about existing datasets.

Is it an official internet standard?

Not yet. It is an individual Internet-Draft submitted to the IETF’s AI Preferences working group, which the IETF has not endorsed.

How does a publisher use it?

By adding a __usage_ai-training group to robots.txt with Disallow lines for the content it wants excluded, and Allow lines for any exceptions.