Free website check

Glossary

GPTBot

Written by
Outseen
Updated

GPTBot is OpenAI’s crawler that collects web content that may be used to train OpenAI’s generative AI foundation models. Blocking GPTBot in robots.txt signals that your content should not be used for that training.

GPTBot is one of the bots OpenAI describes in its overview of OpenAI crawlers. OpenAI uses GPTBot to make its generative AI foundation models more useful and safer: the bot fetches content that may be used for training. GPTBot follows robots.txt, and its token there is GPTBot.

How do you recognise GPTBot?

Your server logs show a user agent like this one (the version number may change):

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

OpenAI also publishes a list of IP addresses for each bot, so you can check whether a request really comes from OpenAI.

Example

User-agent: GPTBot
Disallow: /

Common mistake

Thinking that blocking GPTBot removes you from ChatGPT, or that allowing it makes you visible in ChatGPT. For search in ChatGPT, OpenAI uses a different bot: OAI-SearchBot. OpenAI writes that each setting is independent of the others. You can refuse GPTBot and allow OAI-SearchBot.

More in Allowing or blocking AI crawlers with robots.txt and under AI crawler.

Sources

Read next