A crawler is a program that visits websites on its own, fetching pages and following links to find more. In its overview of crawlers and fetchers, Google calls crawler a general term for any program that automatically discovers and scans websites.
Google divides its own bots into three groups: common crawlers such as Googlebot, which always follow robots.txt when crawling automatically; special-case crawlers, which work under an agreement with the site; and fetchers, which retrieve a page because a user asks for it.
Why does it matter?
What a crawler cannot fetch, a search engine or AI service cannot use. A crawler identifies itself to your server with a user agent. That is how your server logs show which bots visit your site.
Example
A server log line for a visit by Bingbot can look like this (the IP address is an example):
203.0.113.7 - - [11/Nov/2026:09:14:02 +0100] "GET /contact HTTP/1.1" 200 5120 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"The user agent string comes from Microsoft’s overview of Bing crawlers. Because such a string is easy to imitate, you check the IP address separately.
Common mistake
Mixing up crawling and indexing. A crawler fetches a page; whether the search engine then includes it is a separate step. See indexing.
More under AI crawler and in Allowing or blocking AI crawlers with robots.txt.