Google plans to honor a single shared instruction in robots.txt that tells AI companies a website does not want its pages used to train their models. John Mueller, Google Search Advocate, laid out the plan on the Google Search Central Blog. Today a publisher who wants to refuse several AI developers has to write separate rules for each one. The new signal would reach every participating provider in a single step.
The mechanism is a “policy token” placed in robots.txt, the file at the root of a site that crawlers read before fetching pages. It builds on the AIPREF effort inside the IETF, the Internet Engineering Task Force, where a definition of “AI training” was drafted. Google adopts only that definition for now. Talks on the remaining AIPREF terms are still going.
Here is how Google says it will treat the token. If a site uses it to opt out of training and does not mention Google-Extended, Google will treat the site as having blocked Google-Extended, the training control the company has offered since 2023. Other AI providers can handle the token the same way, the post says, which would let a site owner opt out across every provider that adopts it or choose provider by provider.
Nothing existing gets replaced. Company-specific controls keep working next to the shared signal, so a publisher could refuse training broadly and still write a tailored rule for one developer. The post adds that a site with an explicit Google-Extended allow can show up in Gemini Apps answers and collect clicks through grounding. For a publisher, that is the cost-benefit in one sentence: opting out of training is easy, but an explicit allow is how a site stays visible in Gemini answers.
Google frames the proposal as an answer to publishers, advocacy groups, and regulators who asked for controls that span the industry. It also points to the EU’s practice code for providers of broad-purpose AI models, written under the AI Act, which names robots.txt as a main route for publishers to state machine-readable preferences.
The timeline is the catch. This is a draft proposal, and Google says only that it aims to support the control in 2027, together with other AI providers. The post names no other company that has committed, and it does not describe how anyone would verify that a provider honors the signal. A shared opt-out is worth exactly as much as the number of AI developers that agree to read it, and so far the only one on record is Google.
For site owners the consequence is concrete: keep your current Google-Extended and per-crawler rules in place, because the token does not exist in practice until 2027. When it arrives, an explicit Google-Extended allow will be what keeps a blanket refusal from also removing you from Gemini Apps.
Reported by the Google Search Central Blog (Google for Developers), in a post by John Mueller, Google Search Advocate.
