Google, Microsoft, OpenAI and Anthropic Back One AI Training Opt-Out: Websites Need Just One robots.txt Rule

Last week, we wrote about Google starting to pay for content used in AI search. Right after that came news with a wider reach: on October 30, 2026, Google, Microsoft, OpenAI and Anthropic each said they will honor one shared robots.txt signal that lets websites opt out of AI training on their content.

Until now, blocking each company meant remembering every bot name, such as GPTBot, ClaudeBot or Google-Extended. With the new signal, a single group can say "do not use this for training" to several companies at once. It is still a proposal, though, and each company follows its own timeline, so it helps to know what it does and what it does not do.

One robots.txt rule for all four companies

The new signal is a special user-agent group called __usage_ai-training, usually paired with Disallow: /. It comes from an Internet-Draft in the IETF AI Preferences working group, written by Kevin Kelley of Anthropic.

• Syntax: add User-Agent: __usage_ai-training and Disallow: / to robots.txt to say you do not want your content used for AI training.

• It must be explicit: a general User-Agent: * rule does not count as this signal. A site without its own group is treated as having stated no preference.

• Not an official standard yet: this is an individual IETF draft and has not been approved as a standard.

• Voluntary: like every robots.txt rule, its effect depends on each crawler choosing to honor it.

Different timelines, and training is not the same as answering

The most important point is that the four companies are not starting at the same time, and "training" is different from AI reading your page to answer a question (grounding).

• Anthropic: says it has already started.

• OpenAI: will support it in the coming months, and existing rules such as blocking GPTBot will keep working.

• Google: plans support in early 2027. Reports differ on whether the signal affects AI Overviews and AI Mode, so it is worth waiting for Google's official documentation.

• Microsoft: starts on February 1, 2027, as it recrawls each page. Microsoft separates training and grounding preferences, and the NoArchive and NoCache tags will no longer serve this purpose after March 1, 2027.

• Meta: has not committed to honoring the signal.

The new signal also does not remove data already used for past training, and it does not take your site out of regular search results such as Bing.

The Agenorix view: decide first, edit robots.txt second

For most websites, the goal is not to block AI entirely. As people increasingly ask AI instead of clicking through search results, being read and cited by AI is an important way to reach them. That is why we recommend deciding on your goal before changing a single line of robots.txt.

1. Separate three needs: being found in search, being cited by AI in answers, and allowing AI training. These can now be declared separately.

2. Check your current robots.txt: many sites accidentally block search bots when trying to block AI bots, or the other way around.

3. Only block training when you have a reason: paid content, proprietary data or internal documents are the cases worth considering.

4. Log changes and track them: after each edit, watch traffic from search and from AI tools for a few weeks.

Where to start?

Open your-domain/robots.txt and answer three questions: which bots you are blocking, why you block them, and whether you want AI to train on your content. If you need help reviewing your robots.txt and your visibility in AI search, the Agenorix team is ready to help, and your first consultation is free: get in touch.

Sources: Search Engine Journal (Oct 30, 2026), PPC Land (Oct 30, 2026), IETF AI Preferences Working Group, Yahoo Tech

Share