Crawl budget

Crawl budget is the practical limit on how much of a site a search engine will fetch over time. Google describes it as a function of crawl capacity (how much it can fetch without overloading your server) and crawl demand (how much it wants to, based on popularity and staleness). For most small and medium sites it is a non-issue; Google crawls them fully. It becomes real for large sites, sites with millions of URLs, faceted navigation, or heavy parameterization.

Budget is wasted on low-value URLs: infinite parameter combinations, duplicate pages, soft 404s, endless calendar pages, and slow responses that reduce how much the crawler attempts. The remediations are to consolidate duplicates with canonicals, block genuinely useless URL patterns in robots.txt, fix slow responses (faster pages mean more pages crawled), remove or noindex thin pages, and keep the sitemap clean so the crawler spends its budget on URLs that matter.

For AI visibility, crawl efficiency has a second payoff: the AI crawlers also have finite patience, and a site that responds quickly and exposes its content in raw HTML gets read more completely. We surface crawler distribution and last-crawled timing where log data is available, so budget waste can be seen rather than inferred.