How-to steps · Full guide

AI crawlers and your robots.txt

Crawler names change as the AI companies add new ones, which is why this list lives here rather than in print. The rule from chapter 6 holds: never block the crawlers that fetch pages for answers. Blocking training crawlers is a separate business decision.

Last checked 23 September 2026

Who crawls, and what for

Chapter 6

Company	User agent	What it does	Block it?
OpenAI	`OAI-SearchBot`	Finds pages to show in ChatGPT search	No
OpenAI	`ChatGPT-User`	Fetches a page when a ChatGPT user's request needs it	No
OpenAI	`GPTBot`	Collects training data	Your call
Anthropic	`Claude-SearchBot`	Indexes pages for Claude's search results	No
Anthropic	`Claude-User`	Fetches a page when a Claude user asks	No
Anthropic	`ClaudeBot`	Collects training data	Your call
Perplexity	`PerplexityBot`	Indexes pages for Perplexity answers (not training)	No
Perplexity	`Perplexity-User`	Fetches a page when a user asks	No
Google	`Googlebot`	Google Search, including AI Overviews and AI Mode	Never
Google	`Google-Extended`	Not a crawler: a control that opts you out of Gemini training and of grounding in Gemini apps. No effect on Google Search	Your call
Microsoft	`bingbot`	Bing, which also feeds Copilot	No
Apple	`Applebot`	Search in Siri, Spotlight and Safari	No
Apple	`Applebot-Extended`	Not a crawler: a control that opts you out of Apple's AI training	Your call
Meta	`Meta-WebIndexer`	Meta AI's search	No
Meta	`Meta-ExternalAgent`	Training and product improvement	Your call
Common Crawl	`CCBot`	The open web dataset many AI models are trained on	Your call

Fetchers that act for a person who asked (ChatGPT-User, Perplexity-User, Meta-ExternalFetcher) may not follow robots.txt at all; Anthropic says all three of its bots do. Microsoft has no separate Copilot token: its noarchive meta tag keeps a page out of Copilot answers and training.

Opting out of Google's AI answers is a different switch

Chapter 6

Blocking Google-Extended does not remove you from AI Overviews or AI Mode. The switch for that is in Search Console, under Settings, then Search generative AI, or a nosnippet or noindex tag on the page. For almost every business, being in those answers is the point, so leave it alone.

Check your robots.txt in two minutes

Chapter 6

Open yoursite.com/robots.txt in a browser.
Look for a User-agent: line naming any bot marked No or Never above, followed by Disallow: /. That line takes you out of that assistant's answers.
Also check the User-agent: * group: a blanket Disallow: / there blocks everyone.
Re-check after every site release. Staging rules copied to the live site are the classic silent killer.

# Welcome the crawlers that feed answers
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Training crawlers are a separate decision
User-agent: GPTBot
Disallow: /

One-click versions: Hawk Academy's free AI Crawler Access Checker reads your robots.txt against 16 AI crawlers, and the Robots.txt Tester shows which line allows or blocks a given address for a given crawler. Read the results against the table above: only the crawlers that feed answers need to be allowed.

The raw HTML test

Chapter 6

Most AI crawlers read the HTML your server sends and never run JavaScript. Check that your headline and main copy are in that HTML.

Open a terminal and run the command below with your own address.
Search the output for your headline and a sentence from your main copy. Found: fine. Only scripts and empty div tags: the page is invisible to most AI crawlers.
No terminal? Paste the address into Hawk Academy's free Can AI See Your Page? checker, which shows the raw HTML a non-JavaScript AI crawler receives. Or, in Chrome, right-click the page and choose View page source (not Inspect).

curl -A "OAI-SearchBot" https://yoursite.com

On Windows, type curl.exe instead of curl in PowerShell: in Windows PowerShell curl is a different command and the -A option fails.

Some sites send bots different HTML from browsers (pre-rendering, or a firewall rule). The free Pre-rendering Tester requests one address as a browser, as Googlebot and as GPTBot, and shows what differs.