OpenAI, Google and Microsoft have said they will honor a shared signal in robots.txt that lets a website opt out of AI training once, for every participating provider, instead of writing separate rules for each crawler. The signal is a policy token, __usage_ai-training, defined in a draft published on the IETF on October 30, 2026 and written by Kevin Kelley, who works at Anthropic.
The draft adds no new directive and no new syntax. A site writes an ordinary robots.txt group whose User-Agent line names the token instead of a crawler. The draft's own example is User-Agent: __usage_ai-training, Disallow: /, Allow: /press-kit/: no to training on every path except /press-kit/, where it says yes.
Image: OpenAI — https://openai.com/index/helping-websites-express-ai-training-preferences/
Who said what
OpenAI calls it a voluntary effort and says it aims to implement the signal "in the coming months", urging other providers to do the same. Google's crawler documentation says it intends to support it in early 2027. Microsoft is the most specific: from February 1, 2027 it will honor the new preferences as it crawls. Anthropic, whose employee wrote the draft, had no announcement on its newsroom when we checked; according to AI Weekly and aichatdaily it has already begun implementing the signal, but we found no official company statement on the token. Meta signaled support without saying whether it will honor it.
Three rules in the draft
The draft sets three principles. First, a User-Agent: * group does not match the token, so a site that blocks everything with a wildcard has expressed no training preference. Second, if the file has no __usage_ai-training line, no preference is expressed. Third, the token does not control crawling: a provider may still fetch a page it should not train on, and to stop the fetch a site must still name the crawler, as it does today.
Microsoft's extra layer
Microsoft adds two things the draft lacks. Its training preference also works page by page, and a page-level setting overrides robots.txt. It also introduces a separate grounding preference, covering use of a page to build answers in Bing, Copilot and Microsoft 365: Disallow blocks grounding, while the new DisallowExtended blocks it only in services Microsoft sells to others, such as API access. Both default to allowed. The new settings replace the NoArchive and NoCache tags, which Microsoft stops reading as training or grounding preferences after March 1, 2027. Microsoft is plain that neither choice guarantees inclusion, traffic or payment.
Image: cellcog.ai — https://cellcog.ai/blog/ai-training-robots-txt-signal/
What is unsettled
The draft is an individual Internet-Draft: the IETF says it is not endorsed by the IETF and has no formal standing yet. Company-specific tokens, such as a GPTBot group, may take precedence over the shared one, and private agreements may override it. And the signal is not yet live at any provider.
Our case
CellCog publishes a blog and product pages it wants AI engines to read and cite. Its robots.txt has allowed every crawler, training crawlers included, since September 2, 2026, and it has no __usage_ai-training group, so under the draft it expresses no preference. For a site that wants a say, the change is concrete: one group instead of a list of company names, read by three of the largest providers within a few months.
No comments yet. Be the first!