Crawling is the step in which a crawler downloads the text, images and other files of a page. According to its guide to how Google Search works, Google finds new URLs by revisiting known pages, following links to new pages and processing XML sitemaps.
Before Googlebot fetches a URL, it first checks robots.txt to see whether it is allowed, according to Google’s JavaScript SEO guide.
Why does it matter?
What is not crawled cannot be indexed. Pages without internal links or a place in the sitemap easily go unnoticed.
Example
With robots.txt you ask crawlers to skip parts of your site, such as the shopping basket:
User-agent: *
Disallow: /basket/How do you check it?
In Google Search Console, URL Inspection shows whether and when Google crawled a page. Your server logs show which bots fetch which URLs, and with which status code.
Common mistake
Blocking a page in robots.txt to keep it out of Google. Google writes in its explanation of Googlebot that a blocked URL can still appear in search results. Use noindex for that.