AI Training Opt-Out: One robots.txt Rule Now Backed by OpenAI, Google, Microsoft and Anthropic

Key takeaways
– The shared AI training opt-out token is __usage_ai-training: add User-Agent: __usage_ai-training + Disallow: / to robots.txt to opt out of AI training across all four labs.
– Anthropic has already started implementing it; OpenAI says “in the coming months”; Microsoft targets February 1, 2027; Google targets 2027.
– The signal covers training only — it does not block crawling, search indexing, or AI answers that quote your pages.
– It is voluntary and not retroactive: content already collected or trained on is not removed by adding the rule.
– Also this week: Meta delayed its new AI-training terms to November 20, and Canada committed up to C$1.6 billion to AI lab Cohere.

Table of Contents

• What exactly did the four AI labs agree on October 30?
• How do I add the AI training opt-out to my site?
• When will each company actually honor the AI training opt-out?
• What does the AI training opt-out NOT do?
• Also this week: Meta delays its AI-training terms to November 20
• Also this week: Canada commits up to C$1.6 billion to Cohere
• Also this week: Wikimedia weighs in on the NYT vs OpenAI case
• Frequently asked questions

What exactly did the four AI labs agree on October 30?

The four companies agreed to honor one common machine-readable signal: a robots.txt policy token called __usage_ai-training. A site owner writes it where a crawler name normally goes, followed by Disallow: /, and every participating company’s crawler is supposed to read it the same way — as “do not use this site’s content for AI training.”

The token comes from an individual IETF Internet-Draft titled “Machine-Readable Training Preference Signal,” authored by Anthropic standards lead Kevin Kelley and dated October 30, 2026. That detail matters: an individual draft is an early-stage proposal, not a finished standard. The IETF itself notes that an individual draft “is not endorsed by the IETF and has no formal standing in the IETF standards process,” and this draft expires on May 3, 2027 unless it is updated. In other words, the industry agreement is real, but the standard behind it is still a proposal.

Until now, opting out meant maintaining a patchwork: Google-Extended for Google, GPTBot rules for OpenAI, separate directives for Anthropic and Microsoft crawlers. The new AI training opt-out token collapses that into one entry — the first time the major labs have aligned on a single training-preference mechanism.

Skimwire’s October 30 report on the shared robots.txt rule — the story every major AI lab joined the same day.

How do I add the AI training opt-out to my site?

Adding the AI training opt-out signal takes two lines in your robots.txt file, at the root of your site:

User-Agent: __usage_ai-training
Disallow: /

That blocks training use of your whole site. You can also carve out exceptions with Allow: lines — the draft’s own example allows a press-kit folder while disallowing everything else:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Three practical warnings apply before you rely on this AI training opt-out. First, timing is uneven: only Anthropic has started implementing support, so the rule does little at most companies today. Second, keep your existing company-specific rules (like Google-Extended) in place — they are what actually works right now. Third, the token controls training permission, not crawling behavior: as the draft itself states, it “is not a directive for crawling behavior,” so opting out does not stop a crawler from visiting or caching a page.

When will each company actually honor the AI training opt-out?

The AI training opt-out rollout timelines are the weakest part of the announcement, and they differ sharply by company:

Google’s John Mueller confirmed that if a site uses the shared token without naming Google-Extended, Google will interpret it as if the publisher had disallowed Google-Extended. Microsoft went further than the others: it is adding a page-level training setting and a separate control for AI grounding — the retrieval step that pulls live web content into chat answers, rather than training model weights. That distinction matters, because the shared token does not cover AI answers that quote your pages.

What does the AI training opt-out NOT do?

Four limits of the AI training opt-out are worth stating plainly, because the headlines can overpromise:

1. It does not remove anything already collected. Updating robots.txt does not pull content out of existing datasets or trained models.
2. It does not block search. Microsoft says opting out of training does not affect eligibility for regular Bing results, and the token is not a crawl directive.
3. It does not cover AI answers. A page opted out of training can still be retrieved and quoted by an AI assistant answering a question.
4. It has no enforcement mechanism. robots.txt has always been a voluntary convention; the token expresses a preference and relies on each crawler operator honoring it.

None of that makes the announcement empty — a single shared signal is genuinely easier for small publishers than four separate rules — but a site owner who wants protection today still needs the existing company-specific controls, and a site owner who wants old data removed needs a different process entirely.

Also this week: Meta delays its AI-training terms to November 20

On the same day, Meta postponed the enforcement date of its updated commercial terms from October 30 to November 20, 2026. The updated terms expand the scope of content Meta may use for AI training, and the delay followed pushback from news publishers and advertisers, with complaints submitted to antitrust authorities in the European Union and the United Kingdom. For publishers, the two stories point the same way: the rules for who can train on web content are being renegotiated in public, and the next deadline to watch is November 20.

Also this week: Canada commits up to C$1.6 billion to Cohere

On October 29, 2026, Canadian Prime Minister Mark Carney announced the Government of Canada will make an equity investment of up to C$1.6 billion in Cohere, the Toronto-based AI company that builds secure AI systems for businesses and governments. The investment — a commercial equity stake, not a grant — will be made in stages alongside private investors including NVIDIA, Bell, and Germany’s Schwarz Group, inside a Series E round targeting up to C$7 billion at a C$35.7 billion valuation.

In return, Cohere committed to keep its incorporation and headquarters in Canada, create at least 1,000 new high-paying Canadian technical jobs by 2030, and invest at least C$1.7 billion in additional Canadian computing infrastructure by 2030. The deal is one of the clearest examples yet of a government buying directly into an AI developer as a matter of national technology policy — a sovereign-AI bet, rather than a subsidy.

Also this week: Wikimedia weighs in on the NYT vs OpenAI case

Also on October 30, the Wikimedia Foundation submitted an amicus brief to the U.S. District Court for the Southern District of New York in the case between The New York Times and OpenAI — the highest-profile copyright lawsuit over AI training data. The brief adds the operator of Wikipedia, one of the most-crawled reference sites on the web, to a case that will shape how training-data disputes are decided. Read together with the shared opt-out token, the week’s theme is unmistakable: 2026 is the year the web’s content rules for AI get rewritten, in standards drafts, in company policies, and in court.

Frequently asked questions

Does the new robots.txt token replace Google-Extended and GPTBot rules?

No — not yet. Keep your existing company-specific rules in place, because only Anthropic has started implementing the shared AI training opt-out token. The shared token is meant to eventually replace the patchwork, but today the old rules are what actually work.

Will opting out of AI training hurt my Google or Bing rankings?

No. The AI training opt-out token is a training-preference signal, not a crawl directive, and Microsoft has said opting out of training does not affect eligibility for regular Bing results. Search indexing is unaffected.

Can I opt out only part of my site from AI training?

Yes. Use the AI training opt-out block with Disallow: / for the whole site, then add Allow: lines for specific folders, as in the draft’s press-kit example. You can also write the token block alongside your existing per-crawler rules.

Is the AI training opt-out legally binding?

No. It is a voluntary signal based on the long-standing robots.txt convention, and the underlying document is an individual IETF draft with no formal standing. Legal enforceability would come from law or contract, not from robots.txt itself.

Bottom line

October 30, 2026 gave website owners the AI training opt-out they have asked for since AI training disputes began: one rule, honored by all four major labs, instead of four separate ones. The catch is timing — real protection arrives in stages through 2027, the signal is voluntary and training-only, and it never removes what was already collected. Add the two lines now, keep your existing rules until the new token is live everywhere, and watch Meta’s November 20 terms deadline next.

Last checked: October 2026. Sources: OpenAI, Google Search Central, Microsoft Bing, IETF Internet-Draft draft-kelley-aipref-training-preference-signal-00 (October 30, 2026), Prime Minister of Canada (October 29, 2026), and contemporaneous reporting from The Verge, SiliconANGLE, and Search Engine Land. Explore more free tools in our Free Tools Library.

Sources: OpenAI, Microsoft Bing, IETF draft, and Prime Minister of Canada.