ChatGPT, Gemini, Perplexity, Copilot and Claude draw their answers from two sources: what the underlying language model learned during training, and information the service fetches from the web at the moment of the question. For that second source, each service uses its own crawlers or an existing search index. Whether your website can appear as a source therefore depends on different access rules for each service.
The five services side by side
The table is compiled from the providers’ official documentation, checked on 11 November 2026. Where a provider does not describe something, the table says so.
| Service | Crawlers and index | Searches the web live? | Shows source links? | Documentation |
|---|---|---|---|---|
| ChatGPT (OpenAI) | OAI-SearchBot for search, ChatGPT-User for user actions, GPTBot for training. Sometimes works with other search providers. | Yes, automatically when a question needs current information, or when the user chooses Search | Yes, citations and an overview under Sources | Crawlers, search |
| Gemini Apps (Google) | Crawling is done by Google’s existing crawlers. The Google-Extended token controls training and grounding. | Yes, through grounding with Google’s search index | Sometimes; Google says not every answer includes sources | Crawlers, sources |
| Perplexity | PerplexityBot for search results, Perplexity-User for user requests | Yes, in real time | Yes, numbered citations | Crawlers, how it works |
| Copilot (Microsoft) | Bingbot; Copilot builds on the same crawling and indexing foundation as Bing | Yes, through grounding on Bing’s index | Yes, citations | Bing Webmaster Guidelines |
| Claude (Anthropic) | Claude-SearchBot for search, Claude-User for user questions, ClaudeBot for training. Which search index Claude uses is not publicly documented. | Yes, with Claude’s web search feature | Yes, answers that include a search contain citations | Crawlers, search |
What each service says about itself
ChatGPT
OpenAI says ChatGPT searches the web automatically when a question would benefit from current information. ChatGPT search sometimes works with other search providers and then rewrites the question into one or more targeted searches. To be able to appear in the search results, OAI-SearchBot must be allowed to fetch your site and your hosting must let OpenAI’s IP addresses through. OpenAI also says placement is not guaranteed. Covered in detail in How do you become visible in ChatGPT?
How ChatGPT searches the web, according to OpenAI
- QuestionThe user asks something that needs current information, or chooses Search themselves.
- RewriteChatGPT turns the question into one or more targeted searches, sometimes adding a general location or saved memories.
- SearchThe searches go to search providers OpenAI works with, and to pages OAI-SearchBot was allowed to fetch.
- AnswerChatGPT writes the answer with inline citations and an overview under Sources.
Gemini
Google describes grounding in Gemini Apps as providing content from Google’s search index to the model at the moment of the question. Grounding means an AI answer is supported by sources that were fetched for it. Google-Extended has no user agent of its own: Google crawls with its existing crawlers, and the token only states what Google may do with the content. If you block Google-Extended, your content may no longer be used to train future Gemini models or for grounding in Gemini Apps. According to Google, this has no effect on Google Search. We cover the AI features in Google Search itself in AI Overviews and AI Mode.
Perplexity
Perplexity says it searches the web in real time and gives every answer numbered citations. PerplexityBot is meant to show and link websites in the search results and, according to Perplexity, is not used to collect content for AI foundation models. Perplexity-User fetches pages when a user asks a question, and according to Perplexity it generally ignores robots.txt because a user requested the visit.
Copilot
In the Bing Webmaster Guidelines, Microsoft says Bing and Copilot work on the same foundation of crawling, indexing and ranking as ordinary search. According to Microsoft, the same basic rules also help pages qualify for grounding and citations in Copilot. Microsoft also names rules that affect Copilot specifically: NOARCHIVE prevents content from being used in Copilot answers, and NOCACHE limits Copilot to the URL, the title and the snippet.
Claude
Anthropic names three crawlers. Claude-SearchBot improves the quality of search results; blocking it can reduce your visibility in search results. Claude-User fetches pages when a user asks for them. ClaudeBot collects content that may contribute to training models. According to Anthropic, all three respect robots.txt. The help article on search says Claude uses a search feature when current information would help, and that answers which include a search contain citations. Anthropic does not say which search provider is behind it.
Why do they give different answers to the same question?
The differences have several causes:
- A different index. Gemini searches Google’s index, Copilot searches Bing’s, and the other services use their own or combined sources.
- Rewritten searches. OpenAI describes how ChatGPT turns a question into one or more searches and can take a general location and saved memories into account.
- Different models. Perplexity says it uses several language models and that paying users can choose a model themselves.
- With or without a search. Not every answer includes a search. Without one, the answer comes from the model itself.
Say someone in Zwolle puts the same question to two AI assistants. One answer could look like this:
Which bike shop in Zwolle repairs e-bikes on Saturdays?
On Saturdays you can go to Fietsenwinkel De Spaak, among others. According to its website it repairs e-bikes from most brands and is open until 5 pm on Saturdays. Call ahead if the problem is the battery, as not every workshop keeps parts in stock.
Bronnen: despaak.example.com/repairs | bikes-zwolle.example.com
That makes one answer on one platform a snapshot. If you want to know how often a business appears in AI answers, you need repeated measurements with fixed questions. How to set that up is covered in How do you measure your visibility in AI answers?
What do you need to set up on your website?
Three points follow directly from the documentation:
- Manage access per crawler. Blocking a search crawler costs you visibility in that service. Blocking a training crawler does not affect the search feature at OpenAI and Anthropic. According to Perplexity, its two crawlers are not used to train models. At Google, Google-Extended does affect grounding in Gemini Apps. The rules for each crawler are in Allowing or blocking AI crawlers with robots.txt.
- Make sure Google and Bing know your pages. Gemini relies on Google’s index, Copilot on Bing’s. Google Search Console and Bing Webmaster Tools show which pages are indexed.
- Create pages that make sense without context. Microsoft asks for explicit facts and important information early on the page. That advice helps with the other services too, because they all need to be able to use a piece of your page as an answer.
What happens when you block a crawler, according to the providers
| Used for | If you block it | |
|---|---|---|
| OAI-SearchBot (OpenAI) | Search results in ChatGPT | Your site does not appear in ChatGPT’s search answers |
| GPTBot (OpenAI) | Training models | No training on your content; the search feature stays |
| Claude-SearchBot (Anthropic) | Search results in Claude | Your visibility in search results may drop |
| ClaudeBot (Anthropic) | Training models | No training on your content; the search feature stays |
| PerplexityBot | Showing and linking in Perplexity | Perplexity recommends allowing it to appear in search results |
| Google-Extended (token) | Training Gemini and grounding in Gemini Apps | No training or grounding; Google Search is unaffected |
| Bingbot | Bing’s index, also used by Copilot | Bing cannot fetch your pages, and Copilot relies on that index |
This is what a robots.txt looks like that lets the search crawlers in and keeps the training crawlers of OpenAI and Anthropic out:
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /Whether you keep training crawlers out is your own choice. For visibility in the search features, the search crawlers are what count.
How Outseen works with this
The team behind Atlas follows the official documentation of Google, Bing, OpenAI and Anthropic, such as the pages in this comparison. That is not a partnership and gives no knowledge of secret algorithms. Outseen checks by hand how often an AI service mentions your business, with agreed questions, environment and time of measurement.
What does this mean for your business?
Open your robots.txt and check, for each search crawler, whether it may fetch your site: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot. Ask your hosting provider whether your CDN or firewall lets these bots through, because an allowed bot that hits a block still cannot get in. Then check in Search Console and Bing Webmaster Tools that your most important pages are in the index.
Next, put the same customer question to each of the five services and note who is mentioned and which pages appear as sources. Differences between the services are normal. They show where your access or your content still falls short. What you can improve after that is covered in What is GEO?