Google will support a shared AI training control in robots.txt in 2027

Google said on 30 October 2026 that it plans to voluntarily adopt a proposed AI training control for websites. The proposal is under consideration at the IETF and is based on work from its AIPREF working group. Google's crawler documentation says it intends to support the signal in early 2027.

The control lets a site state its AI training preferences for several providers with one robots.txt entry, instead of listing each provider's crawler. Google's documentation writes the signal as user-agent: __usage_ai-training. Google said Bing, OpenAI and Anthropic have made similar moves.

What Google clarified

Google Search Advocate John Mueller explained how it will work alongside existing rules. If a site uses the shared token to opt out of training and does not mention Google-Extended, Google will treat that as a Google-Extended disallow. Company-specific controls such as Google-Extended stay in effect next to the shared signal.

Sites that explicitly allow Google-Extended can still appear in Gemini Apps responses and receive clicks through grounding, according to Mueller's comments reported by Search Engine Roundtable.

The standard is not finished. Nate Hake, who has followed the AIPREF work for about a year, told Search Engine Roundtable that significant disputes remain. His main objection is that the current draft exempts user-initiated actions, which he says hurts creators facing AI copycat spam.

Why it matters

Many publishers now keep a growing list of AI crawler rules in robots.txt, and each provider has its own name. A shared token could make that list shorter and easier to audit.

It also forces a decision that sits between two goals. Blocking training can protect your content. Blocking too much can reduce how often AI assistants know about your brand, and visibility in AI answers now drives real traffic and leads.

Training controls and search access are not the same thing. Google's own documentation treats training and grounding separately, so check which rule affects which product before you change anything.

What to do this week

• Open your robots.txt and list every AI-related rule. Note who added each one and why.
• Decide your policy: allow training, opt out of training, or choose by provider. Write it down so the next change does not undo it.
• Keep Google-Extended explicit. Mueller said an explicit allow keeps your pages eligible for Gemini Apps responses.
• Do not add the new token yet. Google says support starts in early 2027, and the draft may still change.
• Check that your important pages are still readable by AI crawlers that you want to reach. A crawlability audit will show what blocks them.

Google has said it will keep working on the standard and the rest of the AIPREF vocabulary, so expect more updates before the 2027 rollout.