With robots.txt you state for each AI crawler whether it may fetch your website. OpenAI, Anthropic and Google use separate crawlers or tokens for search and for training models. That means you can refuse training and still be findable in their AI search features. robots.txt is, however, a request that well-behaved crawlers follow. It secures nothing, and it guarantees no visit or mention.
What is robots.txt?
robots.txt is a text file in the root of your domain, for example yourdomain.com/robots.txt. The file contains groups of rules. Each group starts with one or more User-agent lines: the name of the crawler the rules apply to. These are followed by Allow and Disallow lines with the paths that are or are not permitted.
Since 2022, the rules have been set out in RFC 9309, a standard from the IETF, the body that publishes internet standards. It also sets out the order in which a well-behaved crawler reads the file. Some of those rules are often misunderstood, especially the first two steps below.
How a crawler reads your robots.txt
- Fetch the fileIf robots.txt returns a 4xx error code, the crawler may fetch everything. With a server error (5xx), it has to assume everything is disallowed.
- Look for its own groupIf the crawler has a group of its own, such as
User-agent: GPTBot, it follows only that group and ignoresUser-agent: *. - Otherwise the general groupIf the crawler is not named anywhere, the rules under
*apply. If there is no such group either, there are no restrictions. - Longest rule winsIf both
AllowandDisallowmatch a URL, the rule with the longest matching path applies. - Fetch or skipWhat is not forbidden is allowed. The crawler then decides for itself whether and when to visit.
In its introduction to robots.txt, Google says the file is meant to manage crawler traffic and is not a way to keep a page out of Google. For that you use noindex or a password.
Which AI crawlers are there?
The table lists the crawlers and tokens that the major AI services document themselves, checked on 11 November 2026. A user agent is the name a crawler identifies itself with, and the name you use to address it in robots.txt.
| User agent | Company | Purpose | Respects robots.txt? | Effect of blocking |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | Showing and linking websites in ChatGPT search | Yes | Your pages do not appear in ChatGPT’s search answers |
| ChatGPT-User | OpenAI | Fetching a page when a user takes an action | Possibly not, because the user asked for it | Does not decide whether you appear in ChatGPT search |
| GPTBot | OpenAI | Content for model training | Yes | A signal that your content may not be used for training |
| Claude-SearchBot | Anthropic | Quality of search results in Claude | Yes | Less visibility and accuracy in search results |
| Claude-User | Anthropic | Fetching a page when a user asks a question | Yes | Claude cannot fetch your pages for user questions |
| ClaudeBot | Anthropic | Content for model training | Yes | Future material is excluded from training |
| PerplexityBot | Perplexity | Showing and linking websites in Perplexity; not for training | Perplexity recommends allowing it in robots.txt | Your pages may be missing from Perplexity’s search results |
| Perplexity-User | Perplexity | Fetching a page when a user asks a question | Generally not, according to Perplexity | Little effect through robots.txt |
| Google-Extended | Token for Gemini training and grounding in Gemini Apps | Yes, it is a robots.txt token without a crawler of its own | No effect on Google Search; does affect grounding in Gemini Apps | |
| Googlebot | Google Search, including AI Overviews and AI Mode | Yes | Google can no longer fetch your pages for Search | |
| Bingbot | Microsoft | Bing and Copilot | Yes | Microsoft advises against blocking Bingbot; less visibility in Bing and Copilot |
Sources: OpenAI, Anthropic, Perplexity, Google and Microsoft. How these services put their answers together is covered in ChatGPT, Gemini, Perplexity, Copilot and Claude: where do their answers come from?
What do the rules look like?
Allow everything
If you have no robots.txt, or only an empty * group, all crawlers may fetch everything. If you want to make that explicit:
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xmlRefuse training, allow search
These rules refuse the training crawlers of OpenAI and Anthropic. The search crawlers fall under the * group and may continue.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: *
Disallow: /cart/Note the effect of the separate groups: GPTBot and ClaudeBot follow only their own group. The rule for /cart/ no longer applies to them, but that does not matter, because they are not allowed anything anyway.
If you also want to refuse Gemini training, add a group for Google-Extended. Weigh that up first: Google uses the same token for grounding in Gemini Apps. Blocking it means Gemini may no longer use your content to support an answer.
Exclude one folder for a search crawler
User-agent: OAI-SearchBot
Disallow: /internal/
Allow: /Here OAI-SearchBot may fetch everything except /internal/. That helps to manage crawler traffic. It does not protect a folder with confidential information; see below.
Limit the crawl rate
Anthropic supports the non-standard Crawl-delay rule, for example Crawl-delay: 1 in the ClaudeBot group. Other providers do not mention that rule in their documentation. So do not assume every crawler follows it.
Refuse training or allow search: which do you choose?
In robots.txt you really make two separate choices: whether your content may be used for training, and whether AI search features may fetch and show your pages. For each goal, it looks like this.
What you want, what you do and what it costs
| What you do | What it costs you | |
|---|---|---|
| As findable as possible in AI search features | Allow search crawlers and let their IP addresses through your firewall or CDN | Nothing except crawler traffic |
| No use for model training | Block GPTBot and ClaudeBot | At OpenAI and Anthropic, no effect on their search features, according to their documentation |
| No use by Gemini | Block Google-Extended | No grounding in Gemini Apps; Google Search is unaffected |
| Not in AI answers at all | Block search crawlers, and use nosnippet for Google | You miss out on referrals from those services |
No choice is right for everyone. An online shop that wants to be found for product questions has little reason to keep search crawlers out. A publisher that earns money from its articles may weigh training differently.
What does robots.txt not do?
- robots.txt secures nothing. RFC 9309 says so plainly: “These rules are not a form of access authorization.” The file is public, so a list of forbidden folders also tells everyone where they are. Use a login for confidential content.
- Not every crawler listens. Google says crawlers decide for themselves whether to follow the rules. Fetchers acting at a user’s request, such as Perplexity-User and possibly ChatGPT-User, may ignore robots.txt.
- IP blocks are not a better opt-out. Anthropic says blocking by IP address does not work reliably, because the crawler then cannot read your robots.txt either.
- Access is not a visit, and a visit is not a mention. An allowed crawler will not necessarily come by. And a page that has been fetched will not necessarily be used in an answer.
How do you check your settings?
You don’t need special software for this. Most checks take nothing more than your browser and the settings of your hosting or CDN.
Check your robots.txt in five minutes
- Open
yourdomain.com/robots.txtin your browser and see whether you get the file or an error page - Look for groups with the names from the table and make sure no search crawler is accidentally set to
Disallow: / - Check in your CDN or firewall whether bot protection stops AI crawlers; OpenAI and Perplexity publish IP lists you can let through
- Search your server logs for the user agents from the table, so you can see which crawlers actually visit
- After a change, wait a day: OpenAI and Perplexity say a change takes about 24 hours to take effect
Managing access is one step. Whether a crawler can read your content also depends on how your pages are built. Read Can AI crawlers read your website? for that. An additional file such as llms.txt is covered in What does llms.txt do for your website?
How Outseen approaches this
Atlas, the research engine developed by Outseen, reads the robots rules and the indexing instructions in code and headers that your website returns publicly. For every finding, Atlas records what it found, in which source, at which URL and when. After a change, Atlas can check again. That check does not tell you whether an AI service will then mention your page.
What does this mean for your business?
For most businesses that want to reach customers, the safe choice is to allow the search crawlers of AI services and make a deliberate decision only about the training crawlers. Start by looking at what your robots.txt says now, and ask your hosting provider whether your CDN blocks AI crawlers by default. Bot protection can stop crawlers even when your robots.txt allows them.
If you have content that must not be public, put it behind a login. robots.txt is not designed for that. To understand how search crawlers deal with your site more generally, see the glossary entry on crawling and the technical SEO checklist.