OpenAI, Google, Microsoft and Anthropic Back One Shared Opt-Out for AI Training

For years, website owners who wanted to keep their content out of AI training sets had to play whack-a-mole, writing a separate robots.txt rule for every AI company’s crawler. That may be about to change. OpenAI, Google, Microsoft and Anthropic have each said they will honor a single, shared “no AI training” signal that a site can set once, in the robots.txt file it already uses to talk to search engines.

The move, announced on Friday, October 30, is voluntary and still months away from working across the board. But it is the clearest sign yet that the biggest AI developers are converging on one set of rules for how publishers say yes or no to having their pages used to build AI models.

What Was Announced

In a short post, OpenAI said it will honor “a common signal” in robots.txt that lets websites state their preferences about the use of their content for foundational generative AI training. The company said it aims to implement the signal “in the coming months,” and that its existing crawler controls will remain in place rather than being replaced.

According to The Verge, the shared “policy token” was proposed by Kevin Kelley, a technical lead for standards at Anthropic, through the Internet Engineering Task Force (IETF), the body that develops many of the internet’s core technical standards. The Verge reports that Anthropic has already adopted the signal, while Google and Microsoft plan to support it next year.

Meta is the notable holdout. Meta communications head Andy Stone told The Verge, “We believe creating a common signal that conveys such preferences to multiple providers at once is an important step,” but the company did not say whether it would honor the token.

How the Signal Works

The proposal, published as an IETF Internet-Draft, deliberately adds nothing new to robots.txt grammar. Instead of naming a specific company’s crawler in the “User-Agent” line, a site names a purpose: the policy token __usage_ai-training. Ordinary Allow and Disallow rules then apply. The draft’s own example opts an entire site out of AI training except for a press-kit folder:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

As Search Engine Journal explains, “AI training” here means using content to change a generative AI model’s learned parameters. Models that only do non-generative work such as classification or ranking fall outside that definition. The publication also notes that the IETF tracker lists the document as an individual draft, which means it is not yet official IETF standardization activity, even though it was submitted to the IETF’s AI Preferences working group.

Each Company Is Implementing It Differently

The common token is shared, but the fine print is not. Search Engine Journal’s breakdown of the companies’ posts shows meaningful differences:

• Google says that if a site opts out with the shared line and does not separately specify Google-Extended, it will treat that as if the publisher had disallowed Google-Extended. That control covers training future Gemini models and grounding in Gemini Apps, meaning the use of content to inform AI answers. Google’s documentation points to early 2027 for support.
• Microsoft is adopting the shared line for training and adding a separate, page-level “AI Grounding” preference, plus a meta tag or HTTP header that can set the training preference for a single page. It will begin honoring the new options on February 1, 2027, and will stop treating the older NoArchive and NoCache tags as training or grounding controls after March 1, 2027. Microsoft says pages will still appear in Bing search results regardless of these choices.
• OpenAI says implementation is coming in the next few months. Search Engine Journal notes that, as of October 30, OpenAI had not said what happens when a site’s shared rule conflicts with its existing GPTBot rule.
• Anthropic, whose standards lead wrote the draft, is the furthest along, and Kelley told the working group the company has started building support.

Why It Matters for Businesses

It lowers the cost of saying no. Today, a publisher, retailer or software company that wants to keep its content out of AI training has to track a growing list of crawler names and keep its robots.txt current as new ones appear. One purpose-based token shifts that burden from the website owner to the AI providers who agree to honor it. For small and mid-sized businesses without a dedicated SEO or legal team, that is a real simplification.

Training and visibility are being separated. The more important shift may be conceptual. Microsoft’s split between training and grounding, and Google’s long-standing distinction between search and Google-Extended, mean companies increasingly can choose to keep content out of model training while still appearing in search results. That is a strategic decision, not a technical one. A brand that wants to be cited when customers ask an AI assistant for recommendations may want to allow grounding even if it blocks training. A paywalled publisher may want to block both.

The details still differ by provider. Because Google ties the shared line to Google-Extended, an opt-out may also affect whether a site’s content can be used to ground answers in Gemini Apps. Companies that care about AI visibility should read each provider’s documentation before flipping the switch site-wide, rather than assuming one line has the same effect everywhere.

It is voluntary and not yet universal. The signal is a stated preference, not a technical barrier. It depends on crawlers choosing to respect it, it is not yet a finished IETF standard, and Meta has not committed to honoring it. Businesses with high-value proprietary content should treat it as one layer of protection alongside licensing terms, access controls and monitoring, not as a replacement for them.

What to Do Now

Nothing breaks if a site does nothing today, since OpenAI, Google and Microsoft have not switched the signal on yet. But this is a good moment for marketing, content and legal teams to agree on a policy: which sections of the site should feed AI training, which should be available for AI answers, and which should stay off-limits. Once that is decided, adding the shared token is a few lines of text. Sites still relying on Microsoft’s NoArchive or NoCache tags for AI control have a firm deadline to plan around in early 2027.

The Bottom Line

A single, shared AI training opt-out backed by OpenAI, Google, Microsoft and Anthropic gives website owners a simpler way to express what they want, and it signals that the AI industry is moving toward common ground rules with publishers. The rollout is staggered through 2027, the implementations differ, and the signal remains voluntary. Still, for any business whose website is part of its competitive advantage, the question of how its content feeds AI is now a decision worth making on purpose.

Edited by Coron @QUE.COM
Website: https://QUE.COM Intelligence
Sponsored by: https://MAJ.COM AI Autonomous

---

Edited by Palawan @QUE.COM
Website: https://QUE.COM Intelligence
Sponsored by: https://MAJ.COM AI Autonomous