NightshiftDirect

Glossary

AI search crawlers

AI search crawlers are the bots AI companies send to read websites. Some gather pages for their assistants' search answers, some collect pages to train AI models, and some fetch one page because a user asked about it.

By NightshiftDirect Research. Published . Updated .

The crawlers behind AI search answers

OpenAI says OAI-SearchBot surfaces websites in ChatGPT's search features, and that sites which opt out of it are not shown in ChatGPT search answers (OpenAI).

Anthropic has two bots on the search side. Claude-SearchBot reads the web to improve Claude's search results, and Claude-User fetches pages when a person asks Claude a question. Anthropic says turning off either one may make your site less visible in search results (Anthropic).

Perplexity says PerplexityBot finds and links websites for its search results, and that this bot does not gather pages to train AI foundation models (Perplexity).

Training crawlers are a separate decision

OpenAI says GPTBot crawls content that may be used to train its models, and that its settings are independent, so a site can allow OAI-SearchBot while disallowing GPTBot (OpenAI). Anthropic says restricting ClaudeBot keeps a site's future pages out of its training data (Anthropic).

Whether to allow training is your call. The block to look for is the one nobody meant: a plugin setting or a copied robots.txt that names every AI bot, search crawlers included, when the aim was only to stop training.

Google-Extended is a robots.txt token

Google-Extended is a token you name in robots.txt to manage whether content Google crawls may be used to train future Gemini models, and whether Gemini Apps may use it to ground their answers. Google says it does not affect a site's inclusion in Google Search and is not a ranking signal (Google).

So blocking it is a decision about Gemini alone. For AI Overviews and AI Mode, Google says a page needs nothing extra: it has to be indexed and able to show a snippet (Google Search Central).

Fetchers that act for one person

The last group fetches a page because someone asked about it. OpenAI says robots.txt rules may not apply to ChatGPT-User, since a user starts those visits (OpenAI), and Perplexity says Perplexity-User generally ignores robots.txt (Perplexity). Anthropic says its bots honor robots.txt, Claude-User included (Anthropic).

So robots.txt cannot be relied on to stop ChatGPT-User or Perplexity-User, and the scan leaves those two out of its count. Claude-User stays in (how the scan works).

How the scan reads your robots.txt

The free scan reports a finding when any of the four search-side bots above is blocked (A1). A block on GPTBot or ClaudeBot shows only as a note, since you may have chosen it (A1T), and so does a block on Google-Extended (A1G). Notes add nothing to the estimate.

What the free scan checks

Questions

Can I block AI training but still let AI search tools read my site?

For OpenAI and Anthropic, yes. Disallow GPTBot and ClaudeBot and leave the four search-side bots allowed; OpenAI says its settings work independently. Google-Extended is different: blocking it keeps your pages out of Gemini Apps' answers as well as Gemini training, though Google Search is unaffected. A rule covers only the bot it names, and only from now on.

If I let these crawlers in, will AI assistants cite my business?

Nobody can tell you they will. Letting them in means an assistant can read your pages; what it says is up to the company that runs it. What you control is access, and whether your details match across your site and its structured data.

Where this fits in our work

We handle this as part of AI search (GEO): Site work that helps ChatGPT, Claude, Perplexity and Google AI features read your site, understand who you are and cite you.

Related reading

Posts that use this term

Related terms

Sources