An AI crawler is a crawler that works for an AI service. In their documentation, the large providers distinguish three kinds, each for a different purpose. Which bot you allow or block in robots.txt therefore decides how a provider may use your pages.
| Purpose | OpenAI | Anthropic | Perplexity |
|---|---|---|---|
| Training models | GPTBot | ClaudeBot | none (PerplexityBot does not train) |
| Search and showing in answers | OAI-SearchBot | Claude-SearchBot | PerplexityBot |
| Fetching at a user’s request | ChatGPT-User | Claude-User | Perplexity-User |
Sources: OpenAI, Anthropic and Perplexity. For AI training and grounding, Google has no separate bot but the robots.txt token Google-Extended.
Why does it matter?
With robots.txt you choose per bot. You can refuse training and allow search. Note that bots fetching a page because a user asked for it do not always follow robots.txt. OpenAI writes that the rules may not apply to ChatGPT-User; Perplexity writes that Perplexity-User generally ignores robots.txt.
Example
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /This refuses training by OpenAI and allows ChatGPT’s search bot.
Common mistake
Blocking all AI bots in one go. That also shuts out the search bots, and with them the chance that an AI assistant links to your pages.
More in Allowing or blocking AI crawlers with robots.txt and under user agent.