Robots policy for Claude
A robots policy does not prove inclusion or a mention in Claude.
Claude can search the web and lists its sources with the answer. We work on pages that answer a question in full and check whether the Claude crawlers may read them.
What you get
Your question
Web search
Pages Claude may read
Answer with sources
Sample data
Claude can search the internet for up-to-date information. According to Anthropic, every answer it gives this way includes citations, so you can check for yourself where the information comes from.1
Claude-SearchBot navigates the web to improve the quality of search results in Claude. Block that bot and, according to Anthropic, your site may be less visible and less accurately represented in those results.2
User-agent: Claude-SearchBot
Allow: /ClaudeBot collects content that could contribute to model training. Block ClaudeBot and, according to Anthropic, your future material is excluded from training. Claude-SearchBot has its own rule. So you can refuse training and still stay readable for search.2
User-agent: ClaudeBot
Disallow: /Claude-User fetches a page when someone asks Claude a question that needs it. Block that bot and Claude cannot retrieve your content for such a question. Claude-User follows robots.txt too.2
Checked on 12 November 2026.
Atlas reads your robots.txt separately for Claude-SearchBot, Claude-User and ClaudeBot and keeps search, user questions and training apart. Then Atlas checks the pages Claude could show as a source.
Meet AtlasWe find out which questions customers ask an assistant before they buy, and which pages are missing for them.
We build pages with concrete facts: sizes, materials, delivery time and terms. An answer should be able to take a correct sentence from them, with your page as the source.
Atlas checks whether Claude-SearchBot and Claude-User may read and whether titles, content and product data are in the HTML.
We do not check Claude answers by hand at the moment. Claude is not in any plan, nor in our free scans. You do get the check of the Claude crawlers and the work on your pages.
Claude is not in any plan right now: we do not check Claude answers by hand. We do the work on your pages and the technical setup in every plan, and Claude can read those pages too if Claude-SearchBot is allowed.
Compare the plans| Plan | Claude | Checked by hand | Price |
|---|---|---|---|
| Rise | Not in this plan | 10 answers a month | €549 a month |
| Scale | Not in this plan | 25 answers a month | €949 a month |
| Lead | Not in this plan | 50 answers a month | €1,649 a month |
Excl. VAT, on a three-month term.
Answers to the questions business owners ask us most.
Anthropic names three. Claude-SearchBot navigates the web for the search results in Claude. Claude-User fetches a page when a user asks Claude something that needs it. ClaudeBot collects content that could contribute to model training. All three follow robots.txt, so you set each one with its own rule.
Yes. Training and search run through different bots. With a rule that blocks ClaudeBot, your future content stays out of training, according to Anthropic. Claude-SearchBot and Claude-User each have their own rule, which you can leave in place. If you do block those two, your site may be less visible in Claude, according to Anthropic.
When Claude searches the web, the pages it used are listed as citations with the answer. Claude decides which ones. Pages that answer a question fully and verifiably, with the key text in the HTML and a robots.txt that allows Claude-SearchBot, stand a better chance. Nobody can guarantee a mention.
That is not in Anthropic’s public documentation. What is known is that Claude-SearchBot navigates the web to improve the search results in Claude. A good position in one search engine does not cover Claude automatically. Make sure Claude-SearchBot may read your pages.
No, not at the moment. Claude is not in any of the plans, and our free scans do not measure Claude either. We do use Atlas to check whether the Claude crawlers may read your pages, and we work on the content Claude can use as a source.
We advise against it. According to Anthropic, blocking by IP address may not work correctly and does not reliably keep you out. It can also stop the bots from reading your robots.txt. Use robots.txt instead, with a rule per bot. If you only want to slow the visits down, Anthropic also follows the Crawl-delay rule.