AI & automationWebsites

How to Choose Whether AI Can Train on Your Business Website

Google, Microsoft and OpenAI plan to honor one robots.txt rule that opts a site out of AI training. How a small business decides and adds it, step by step.

By Cybercave Technologies, LLC · Nov 2, 2026

Close-up of a perforated server front panel lit in blue in a dark data center — A server in a data center. Illustration only, not equipment from any company named here. Credit: Photo by panumas nikhomkhai via Pexels (Pexels License).

Short answer: A website can now use one short rule in its robots.txt file to say it doesn't want its pages used to train AI models. Google, Microsoft and OpenAI said on Oct. 30, 2026 they plan to honor that rule, but none of them reads it yet. Decide what you want first, and keep search engines allowed either way.

Key points

On Oct. 30, 2026, Google, Microsoft and OpenAI each announced plans to honor a shared robots.txt rule group, named "__usage_ai-training", that tells AI providers whether a site's content may be used to train generative AI models.
The rule comes from an Internet-Draft written by Kevin Kelley of Anthropic and filed with the Internet Engineering Task Force (IETF), the group that sets many internet standards. It is a proposal, not a finished standard.
Timing differs, as of Nov. 2, 2026: Microsoft starts Feb. 1, 2027, Google plans support in 2027, OpenAI says "in the coming months," and Kelley wrote that Anthropic has started building support.
Opting out of training does not remove a site from Google or Bing search results, both companies say. It also doesn't stop crawlers from visiting.

What to do this week: type your website address followed by /robots.txt into a browser, see what the file says today, and decide whether you want AI companies to train on your pages.

What changed on Oct. 30?

Three major AI providers agreed to read one shared opt-out instead of a separate rule for each company. Until now, a site that wanted to stay out of AI training had to name each company's crawler or control, such as OpenAI's GPTBot or Google's Google-Extended, one by one.

Google said in its Search Central blog that if a site uses the shared rule to opt out and doesn't mention Google-Extended, Google plans to treat it as if the site had blocked Google-Extended. OpenAI said its existing crawler controls will keep working alongside the new signal. Search Engine Journal reported that, based on the timelines each company gave, none of them supported the rule yet when it was announced.

Microsoft went further. Its Bing blog post adopts the shared training rule and adds a separate page-level setting for "grounding," meaning whether a page can be used to help write AI-generated answers in products such as Copilot. Microsoft says sites that rely on its older NoArchive or NoCache tags need to switch to the new settings by March 1, 2027, after which those tags no longer count as training or grounding preferences.

What does the new rule do, and what doesn't it do?

It states a preference about how your content is used. It is not a lock. The IETF draft spells out several limits that matter for a small business:

It doesn't stop visits. Under the draft, the token isn't a crawling rule. To limit which bots visit, you still need rules for each crawler by name.
"Block everything" rules don't count. A rule written for all bots ("User-agent: *") is not read as an AI training preference. Only a group that names "__usage_ai-training" counts.
No rule means no preference. If your file doesn't include the group, you haven't said anything either way. Microsoft says training is allowed by default when no preference is published.
Private deals can override it. The draft says private agreements may take precedence over the robots.txt preference.
It looks forward. Microsoft says its settings cover future AI model training and don't create any licensing or payment arrangement.

Should a small business opt out of AI training?

It depends on what your website is for, and it's your call. Most small-business sites exist to be found, and the training opt-out is separate from search: Google says Google-Extended has no effect on whether a site appears in Google Search, and Microsoft says its new settings don't change eligibility for Bing search results.

The bigger trade-off is AI answers. Microsoft warns that limiting grounding will reduce a brand's visibility in AI-generated answers, and says neither choice guarantees inclusion, traffic or payment. At Google, Google-Extended covers both Gemini training and grounding in Gemini Apps, so Search Engine Journal notes that Google's documentation describes no way to block Gemini training but allow Gemini Apps to draw on your pages for answers.

Our example (hypothetical): a Saugatuck bed-and-breakfast wants its rates and booking page quoted when travelers ask an AI assistant for places to stay, but it doesn't want its hand-written room descriptions used to train models. It could leave search and grounding alone and add the training opt-out only, then check in 2027 how each company actually handles it.

How do I add the AI training opt-out to my website?

If you decide to opt out, it's a short text change to one file. Make a copy of the current file before you change anything.

Find your robots.txt file. It lives at the root of your site, such as www.example.com/robots.txt, and a site has only one, according to Google's robots.txt guide. If nothing loads, your site may not have one yet.
Read what's already there. Look for rules that block Googlebot or Bingbot, or a "Disallow: /" under "User-agent: *". Those affect search, not just AI.
Add the training group. To opt the whole site out, add two lines: "User-Agent: __usage_ai-training" and below it "Disallow: /". To keep one folder available, such as a press page, add an "Allow:" line for that folder, as the draft's own example does.
Keep the company-specific rules for now. Since Google, Microsoft and OpenAI don't read the shared rule yet, existing rules such as "User-agent: GPTBot" (see OpenAI's crawler page) or "User-agent: Google-Extended" (see Google's crawler list) are what currently express a training choice to those companies.
Don't block search by accident. Leave Googlebot and Bingbot allowed unless you truly want to leave search results.
If you can't edit the file, ask. If your site runs on a website builder or hosted platform, you may not be able to edit the file yourself. Google's guide says to contact your hosting provider if you can't reach your site's root.

What should you tell your web provider?

Ask three things in writing: what your robots.txt says today and who can change it; whether they can add the "__usage_ai-training" group if you choose to opt out; and, if your site uses Microsoft's NoArchive or NoCache tags, whether they'll move you to Microsoft's new training and grounding settings before March 1, 2027.

For your IT provider: policy token "__usage_ai-training" (draft-kelley-aipref-training-preference-signal, expires May 3, 2027); Microsoft page-level settings (Bingbot meta tag or X-Robots-Tag header) "ai-training:disallow", "ai-grounding:disallow" and "ai-grounding:disallowextended", honored from Feb. 1, 2027; product tokens GPTBot and Google-Extended still apply.

What should you do this week?

Open your site's robots.txt file and make sure it isn't blocking search engines by mistake. Then decide, with whoever runs your website, whether to add the AI training opt-out now or wait until the companies start honoring it in 2027.

For more, see our guides to checking the third-party scripts on your website, locking down your business domain, and checking who can see what you type into AI.

FAQ

What is a robots.txt file? It's a plain text file at the root of a website that tells automated visitors, such as search engine crawlers, which parts of the site they may visit. The robots.txt standard, RFC 9309, says these rules "are not a form of access authorization," so they depend on bots choosing to follow them.

Does an AI training opt-out lower my Google ranking? Google says Google-Extended, the control the shared opt-out maps to, has no effect on whether a site appears in Google Search and isn't used for ranking. It can affect whether Gemini Apps use your pages in answers.

Does this remove my site from AI models that were already trained? No. The preference applies going forward, and Microsoft says its settings cover future training.

Does every AI company have to follow it? No. OpenAI calls it a voluntary effort. Google, Microsoft and OpenAI have announced plans to honor it, and Anthropic's Kelley wrote that Anthropic has started building support; other companies may not.

Disclosure: No company or organization named in this article paid for it, reviewed it or saw it before publication. Cybercave Technologies builds, hosts and manages websites for small businesses and offers AI automation services. Dates and plans are as published by each source and were checked on Nov. 2, 2026. Not sponsored.