Want to access all premium content?
Become a paying subscriber to get access to all courses, guides like this, prompts and skills library, and $500+ worth of AI tools.
Your website’s robots.txt tells cooperating bots which pages they can crawl. A proposal published on 30 October 2026 adds a shared way to express an AI training preference, using one two-line group rather than separate instructions for every company.
Support is rolling out at different times, so the shared group belongs alongside existing crawler rules. It expresses a request about training; it cannot force a company to comply.
We’ll use Claude to review your file, prepare the additions, and check the published version. You’ll need access to your website’s settings or help from whoever maintains it, plus Claude with web search available.
Step 1: Check your file. Ask Claude to review your existing rules.
Step 2: Pick what stays open. Decide on folder exceptions and Gemini access.
Step 3: Add the group. Place the shared preference in its own group.
Step 4: Keep the old blocks. Retain the crawler-specific rules used today.
Step 5: Check it worked. Inspect the live file and review it again.
Step 1: Check your file
Open your website address with /robots.txt added, such as https://yoursite.com/robots.txt.
If a file exists, you’ll see plain text containing bot instructions. Groups begin with a User-Agent line.
Open Claude, enable web search, and send this prompt with your website address:
Open https://YOURSITE.com/robots.txt and copy the whole file back to me, word for word. Then tell me: 1. Is there a group that starts with User-Agent: __usage_ai-training? 2. If there is, is the next line Disallow: /, with a blank line above the group? 3. Which of these have their own group with Disallow: / : GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot? 4. Is there a Disallow: / line under User-Agent: * ? If yes, warn me, because that hides my site from search engines too. If you can't open the page, say so. Don't guess.
Check that Claude reproduces the file and answers each question. If it cannot retrieve the page, paste the file from your browser and ask the same questions.
If Claude doesn’t recognize __usage_ai-training, that alone doesn’t establish whether the proposal is valid. Its answer may lack information about the recent draft.
Claude reports | What to do next |
No file found | Ask your platform provider or developer how to add one |
Only User-Agent: * groups | Check the existing rules; the draft does not treat a wildcard group as the shared training preference |
__usage_ai-training followed by Disallow: / | The shared preference is present; continue to Step 4 |
The prompt’s search warning needs care: blocking crawling can affect search visibility, but it does not necessarily remove pages already indexed.
Step 2: Pick what stays open
The proposed group uses familiar robots.txt directives. Disallow: / expresses a preference covering the whole site. An Allow line creates an exception, such as a press folder:
User-Agent: __usage_ai-training Disallow: / Allow: /press/
Decide whether you want that exception before preparing the file.
Also consider Gemini. Under Google’s planned mapping, the shared preference acts like a Google-Extended block, which also affects use in Gemini app answers. If you want to retain that access, the supplied guidance calls for an explicit Google-Extended Allow group in Step 4.
Google Search and Bing Search are separate from these training preferences. Preserve their existing access rules.
Step 3: Add the group
Save a copy of your current file before editing it.
Place the two new lines in a separate group at the bottom, with a blank line above them. Preserve the existing rules.
Paste your current file into this prompt:
Here is my current robots.txt: [paste your file] Add this group at the bottom, after a blank line: User-Agent: __usage_ai-training Disallow: / Keep every other line exactly as it is. Don't add Disallow: / under User-Agent: *. Give me the full new file in one code block, then list the lines you added.
If you chose a folder exception in Step 2, include the corresponding Allow line in your request.
Compare Claude’s version with your saved copy before publishing it.
Do not add Disallow: / beneath User-Agent: * for this task. That combination asks all cooperating crawlers, including search crawlers, to avoid the entire site.
Platform | Where to look |
WordPress with Yoast SEO | Open Yoast SEO → Tools → File editor. If available, choose Create robots.txt file when needed. If the editor is missing, ask your host |
Wix | Open SEO & GEO → Tools and settings → Robots.txt Editor → View File, then save your changes |
Shopify | Open Online Store → Themes → … → Edit code and find or create the robots.txt template. Shopify recommends developer assistance |
Squarespace | Direct editing is unavailable. Settings → Crawlers → Block known artificial intelligence crawlers controls named bots rather than adding the proposed group |
A site maintained in code | Follow Use case #2 |
Menu names may differ from those shown here.
If someone else maintains your website, use the developer-request prompt in Use case #3.
Step 4: Keep the old blocks
The supplied rollout information says Microsoft plans support from 1 February 2027, Google targets 2027, and OpenAI describes support as arriving in the coming months. Anthropic says it has begun building support.
Those announcements do not establish that every provider already honors the shared group. Keep the named rules alongside it.
Company or service | Crawler directive | What it controls |
OpenAI | GPTBot | Training crawling; ChatGPT search uses the separate OAI-SearchBot |
Anthropic | ClaudeBot | Anthropic’s crawler |
Google-Extended | Gemini training and Gemini app answers, separately from Google Search | |
Apple | Applebot-Extended | AI training use, separately from search ranking |
Common Crawl | CCBot | Crawling for Common Crawl’s public web archive |
Continue in the same Claude chat:
Take the robots.txt you just gave me. Add one group each, after a blank line, for GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot, each with Disallow: /. Keep every other line exactly as it is and return the full file in one code block.
If you chose to preserve Gemini access, ask Claude to use Allow: / for Google-Extended instead. This also leaves the uses controlled by Google-Extended open.
Review the complete file before saving. Keep these named groups after 2027 as well, since some services may continue relying on their individual directives.
Step 5: Check it worked
Reopen https://yoursite.com/robots.txt and inspect the published text.
Look for the shared group and each named group. Confirm that your earlier rules remain intact.
Run the Step 1 prompt in a fresh Claude chat. Compare its response with the live file, including any intentional Google-Extended exception.
What you see | What to check |
The previous version | A cached copy may still be appearing; wait briefly and reload |
usage-ai-training or a single leading underscore | Use the exact token __usage_ai-training, beginning with two underscores |
New directives mixed into the wildcard group | Put the shared preference beneath its own User-Agent line, separated from the preceding group |
A file at /blog/robots.txt | Publish it at /robots.txt at the root of the relevant host |
A subdomain such as shop.yoursite.com needs its own robots.txt.
Use cases
Use case #1: Review several websites
Ask Claude to check the sites you maintain:
Open robots.txt on each of these sites: https://SITE1.com https://SITE2.com https://SITE3.com Make a table with one row per site and these columns: has a User-Agent: __usage_ai-training group with Disallow: / (yes/no), named groups found (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot), and any Disallow: / under User-Agent: * (yes/no). If you can't open a file, write "could not open" in that row. Don't guess.
Open a sample of the files yourself and compare them with the table. A failed retrieval should remain marked as unknown.
Use case #2: Change the background behind your pet
For a website maintained in code, open its project in Claude Code.
In Terminal, type cd followed by a space, drag the project folder into the window, and press Enter. Then run claude.
Send:
Find this project's robots.txt setup. It's usually public/robots.txt, or app/robots.ts in a Next.js project. Add a group for the user agent __usage_ai-training with Disallow: /, plus groups for GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot, each with Disallow: /. Keep every existing rule. Show me the diff before you save, then tell me which URL to check after I deploy.
Review the proposed changes before approving them. Confirm that existing directives are preserved and that any Gemini exception matches your decision.
Approval behavior depends on your Claude Code settings. The prompt requests a review before saving; check the behavior shown in your session.
Use case #3: Draft a request for your developer
Use this prompt to prepare an email for whoever maintains your site:
Write a short email to [name], who runs my website. Ask them to add these groups to the bottom of our robots.txt, each after a blank line: User-Agent: __usage_ai-training Disallow: / User-Agent: GPTBot Disallow: / User-Agent: ClaudeBot Disallow: / User-Agent: Google-Extended Disallow: / User-Agent: Applebot-Extended Disallow: / User-Agent: CCBot Disallow: / Explain in one line that this asks AI companies not to train on our site and doesn't change Google or Bing search. Ask them to reply once the lines show at [yoursite.com]/robots.txt. Plain words, under 120 words.
Check every directive in the draft, especially the two leading underscores. Adjust Google-Extended if you chose to preserve Gemini access.
Use case #4: Review search crawler access
Ask Claude to identify rules that restrict search crawling:
Open https://YOURSITE.com/robots.txt. List every group that stops Googlebot, Bingbot or OAI-SearchBot from reading any part of the site, including any Disallow line under User-Agent: *. For each one, tell me which pages it hides from Google search, Bing search or ChatGPT search. If you find none, say so. If you can't open the file, say so. Don't guess.
Compare the answer with the file. A restriction on /admin/ may be intentional; a wildcard Disallow: / requests a site-wide crawling block.
Treat any claim that a rule “hides” a page from search cautiously. Crawl restrictions and removal from search results are different outcomes.
Limitations
TThe shared group expresses a preference about training use. It does not prevent access, and a service that disregards it may still collect your pages.
It also cannot undo previous collection or remove information from models already trained on it.
The proposal comes from an individual IETF draft by Anthropic’s Kevin Kelley. It is not a completed standard. The supplied draft lists an expiry date of 3 May 2027 unless renewed.
The supplied guidance also says Bing’s older NoArchive and NoCache tags stop serving as AI preferences after 1 March 2027.
Google’s planned handling links the new group to Google-Extended, so consider the effect on Gemini app answers when choosing your rules.
Claude’s review is an aid, not proof of provider compliance. Inspect the live file yourself and distinguish a correctly published preference from a company’s decision to honor it.
The original guide records a check date of 1 November 2026. Provider timelines, draft details, and platform menus may change.
Closing
Back up your current file, review it with Claude, and add the shared preference alongside the named crawler rules. Then inspect the published version and run the check again.
Repeat the audit for other sites you maintain. Recheck Microsoft’s support around its announced 1 February 2027 start date.
See you in the next one :)