Crawl budget is the amount of crawling Google spends on a website. Google bases it on two things: crawl capacity, meaning how many requests your server can handle without being overloaded, and crawl demand, meaning how much Google wants to crawl your content based on its size, how often it changes and how popular it is. Google describes this in Optimize your crawl budget.
Who needs to care?
Mainly large websites. As a rough guide, Google mentions sites with more than a million pages that change weekly, sites with more than ten thousand pages that change daily, and sites with many URLs listed in Search Console as discovered but not yet indexed. If your new pages are crawled soon after publication, Google says an up-to-date XML sitemap is usually enough.
What wastes crawl budget?
- duplicate URLs with the same content;
- filtered or differently sorted versions of pages;
- soft 404s and removed pages that do not return a 404 or 410;
- long redirect chains;
- slow responses and server errors.
Example
Say an online shop with thousands of products gives every filter combination its own URL. Google then spends time on variants such as ?colour=red&size=m&sort=price instead of on new products. How to manage that is explained under faceted navigation and in Category pages that get found and help people choose.