What is crawl budget?
Crawl budget is the number of URLs a search engine is willing and able to crawl on your site within a given time. It depends on your server's capacity and how much search engines want to crawl your content.
Last reviewed .
Who needs to worry about crawl budget
Google has said that crawl budget is mainly a concern for very large sites, sites with many frequently changing pages, and sites with lots of newly discovered URLs. For most small and mid-sized sites, search engines crawl everything that matters without trouble.
What wastes crawl budget
Duplicate pages, endless URL variations from filters or parameters, long redirect chains, and broken links all give crawlers low-value pages to visit. Slow server responses can also reduce how much gets crawled.
When crawlers spend time on those, important pages may be visited less often or discovered later.
How to use it well
Keep your XML sitemap current, fix broken links and redirect chains, and consolidate duplicate pages. Use robots.txt to keep crawlers out of areas that offer no search value.
The Crawl Stats report in Google Search Console shows how often Googlebot visits your site.
Common questions
What determines crawl budget?
Google describes it as a mix of crawl capacity, meaning how much your server can handle, and crawl demand, meaning how much it wants to crawl your content. Popular, fresh pages generally get more demand.
Does crawl budget affect small websites?
Usually not. Small sites are typically crawled fully, so other issues like content quality and indexing are more likely to hold them back.
How can I check my crawl budget?
Google Search Console's Crawl Stats report shows requests, response times, and crawl activity over time. Server log files give a more detailed picture.
Does blocking pages in robots.txt save crawl budget?
It stops crawlers from requesting those pages, so they spend time elsewhere. A blocked page can still be indexed if other sites link to it, so robots.txt isn't a way to remove pages from search.
Want help putting this to work?
Tell us where your marketing stands today, and we'll show you what to do first.