AI Training Opt Out: One robots.txt Line for 4 AI Firms

Key takeaways

On October 30, 2026, OpenAI, Google and Microsoft said they plan to honour one shared robots.txt group, __usage_ai-training, that tells AI companies you do not want your pages used to train generative AI models.
The idea comes from an individual IETF draft written by Kevin Kelley of Anthropic; it is not an adopted standard and it records a preference, not a crawl block.
Timing differs: Microsoft from February 1, 2027, Google in 2027, OpenAI in the coming months; reports differ on how far Anthropic has got.
At Google, the shared opt out also counts as blocking Google-Extended, which affects grounding in Gemini Apps, but Google says Google-Extended does not affect Google Search.
A robots.txt file without the new group expresses no preference, so nothing changes for your site unless you add it.

A single ai training opt out line for robots.txt is on its way. On October 30, 2026, OpenAI, Google and Microsoft each said they plan to honour a shared signal that lets a website tell several AI companies at once that it does not want its pages used to train generative AI models (OpenAI, Google Search Central, Bing). The Verge reports that Anthropic, whose standards lead wrote the proposal, has also signed on (The Verge).

For anyone running a brokerage, agency or local business website, this is the first time the largest AI companies have lined up behind one switch instead of a separate bot name each. People noticed: in our count of X posts that mention robots.txt or an opt out alongside AI training, daily volume went from between 14 and 17 posts on October 26 to 29 to 110 on October 30 and 489 on October 31 (UTC days). Here is what the line does, what it does not do, and how to decide whether you want it.

What the new robots.txt line looks like

A made-up user agent called a policy token

The proposal is an Internet-Draft by Kevin Kelley of Anthropic for the IETF AI Preferences working group (IETF). It adds no new commands to robots.txt. You write an ordinary group under a reserved name, `__usage_ai-training`, using the familiar Disallow and Allow lines. The draft's own example blocks training on the whole site except a press kit folder:

`User-Agent: __usage_ai-training`
`Disallow: /`
`Allow: /press-kit/`

The draft defines AI training as using content "to modify the learned parameters of a generative AI model", so ranking and classification systems that do not generate content are outside it (IETF).

A preference, not a lock

The draft says the token is not a directive for crawling. Disallow states a preference against training, Allow states a preference in favour, and a file with no such group expresses no preference at all. Rules under `User-Agent: *` do not count, and a private agreement between a site and an AI company may override the signal (IETF). The Verge describes the companies as agreeing to restrict their crawlers when they see the token, but the draft text itself treats it as a statement about use (The Verge).

It is also early. Search Engine Journal notes this is an individual draft, not an official IETF standard, and that none of Google, Microsoft or OpenAI supports it yet (Search Engine Journal). The draft expires on May 3, 2027 unless it is replaced (IETF).

When each AI company says it will honour it

Company	What it said	When
OpenAI	Will honour the signal for foundational generative AI training; existing crawler controls stay	"in the coming months" (OpenAI)
Google	Treats the opt out as if you had disallowed Google-Extended, unless you set Google-Extended yourself	Aims to support it in 2027 (Google)
Microsoft	Reads it for training and adds separate page-level grounding settings	Honoured from February 1, 2027 (Bing)
Anthropic	Wrote the proposal	Already adopted, per The Verge; development started but will take time, per Search Engine Journal
Meta	Communications head Andy Stone signalled support	Did not say it will honour it (The Verge)

The trade-off: training versus being cited by AI

Google: one switch also covers Gemini answers

Google's John Mueller wrote that a site using the shared opt out, without its own Google-Extended rule, is treated "as if the publisher had disallowed Google-Extended". Google adds that sites allowing Google-Extended "are eligible to appear in responses and receive clicks via grounding in Gemini Apps" (Google Search Central). Search Engine Journal points out that Google-Extended does not affect inclusion in Google Search, and that AI Overviews and AI Mode have a separate control (Search Engine Journal).

Microsoft: training and grounding are split

Microsoft is retiring its older NoArchive and NoCache tags. Training can be set in robots.txt or per page, for example with `<meta name="Bingbot" content="ai-training:disallow">`. Grounding, meaning use of your page to answer questions in Copilot and Bing AI features, is set per page only, with Allow as the default (Bing). Microsoft warns that turning grounding off "will significantly reduce your content's visibility in AI-generated experiences", says Bing search eligibility does not change, and says the settings do not create any licensing or payment arrangement. Old tags stop being read after March 1, 2027, and a temporary setting in Bing Webmaster Tools opens on December 15, 2026 for sites that use them.

AI agents visiting for a person are different

OpenAI's crawler page says ChatGPT-User visits pages when a person asks ChatGPT something and, "because these actions are initiated by a user, robots.txt rules may not apply" (OpenAI). So the training line is not a way to stop a buyer's AI agent from reading your listing page, and for most agents that is good news.

What we did not verify

None of the three big providers reads the token yet, so we could not test it. We did not see Anthropic's own announcement or Kevin Kelley's mailing list message, so the Anthropic status rests on The Verge and Search Engine Journal, which describe it differently. The draft ties each preference to the robots.txt file in force when a page was fetched (IETF); no company we read promised to remove content already used.

What this means for small businesses

AI answers are turning into a way buyers, sellers and customers find you, so treat this as a business decision, not a reflex.

Look before you change anything. Open yoursite.com/robots.txt and note what is there. Our AI scraper blocking prompt walks you through which bots you allow without hurting Google rankings.
Decide by section, not site-wide. You might want neighbourhood guides and market reports cited, while keeping paid courses or client resources out of training. Path rules like the draft's press kit example make that possible. Ask your web person to do it, or try our website update prompt.
Remember the Gemini link. At Google, a blanket opt out also turns off grounding in Gemini Apps unless you allow Google-Extended separately.
Diary two Bing dates. If your site uses NoArchive or NoCache, check Bing Webmaster Tools after December 15, 2026 and update tags before March 1, 2027.
Make what you publish worth citing. Write original listing copy with the listing description prompt builder, plan articles with the SEO article prompt, connect pages with our internal linking prompt, and check your prompts with the prompt grader.

Frequently asked questions

How do I opt out of AI training on my website?

Once providers support it, add a robots.txt group for `__usage_ai-training` with `Disallow: /`, as in the draft's example (IETF). Existing controls such as Google-Extended and GPTBot keep working (Google, OpenAI).

Will opting out of AI training hurt my Google rankings?

Search Engine Journal says Google-Extended, which the shared opt out maps to, does not affect inclusion in Google Search (Search Engine Journal). Microsoft says its new settings do not change Bing search eligibility (Bing).

Is __usage_ai-training an official standard?

No. It is an individual IETF draft, not an adopted standard, according to Search Engine Journal, and the draft expires May 3, 2027 (IETF).

Does it stop AI agents like ChatGPT visiting my site?

No. The draft says the token is not a crawling directive, and OpenAI says robots.txt rules may not apply to ChatGPT-User visits that a person triggers (OpenAI).

When does the shared opt out take effect?

Microsoft says February 1, 2027, Google aims for 2027 and OpenAI says the coming months (Bing, Google, OpenAI).

Sources

How we report: every fact, number and quote above comes from the linked sources, which we read before publishing. Product details can change quickly during previews, so check the vendor page before you buy. Spotted an error? Tell us via the link on our home page.

AI training opt out: OpenAI, Google and Microsoft back one robots.txt line