A user agent is the software that sends a request to your server, and the User-Agent header is how it identifies itself. In the HTTP standard RFC 9110, User-Agent is a header with information about that software. Its value consists of one or more product names with an optional version number, sometimes with a comment in brackets. Servers use it for things like statistics and tracing problems with specific software.
User agents in robots.txt
In robots.txt, the User-agent: line holds a short token, not the full header. For Googlebot it is Googlebot, for OpenAI’s training bot GPTBot. A token does not always belong to a bot of its own: according to Google, Google-Extended is a token with no separate user agent in HTTP requests.
Example
This is how GPTBot identifies itself in a request (the version may change):
User-Agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbotAnd this is how you address it in robots.txt:
User-agent: GPTBot
Disallow: /internal/Common mistake
Trusting a visit because the user agent says Googlebot. Google writes that Googlebot’s user agent is often imitated. Check the IP address, for example with a reverse DNS lookup or the IP lists that providers publish.
More under AI crawler and in Allowing or blocking AI crawlers with robots.txt.