What is crawling in SEO?
Crawling is the process search engines use to discover web pages. Automated programs called crawlers, or bots, follow links from page to page and fetch content so it can be analyzed and possibly added to the search index.
Last reviewed .
How crawling works
Crawlers start with known URLs, such as pages they have seen before and those listed in sitemaps. They follow the links they find on each page to discover more.
Crawling is the first step. After a page is fetched, the search engine decides whether to index it, which is a separate process.
Why pages don't get crawled
Pages with no links pointing to them can be hard to find. Blocks in robots.txt, server errors, slow responses, and login walls can also stop crawlers.
Content that only appears after complex scripts run may be harder to process too.
How to help crawlers
Link important pages from other pages on your site, keep an up-to-date XML sitemap, and make sure your server responds reliably. Google Search Console's URL Inspection tool shows how Google sees a specific page.
Common questions
What is a web crawler?
A web crawler is a program that automatically visits web pages and follows links to discover more. Google's crawler is called Googlebot.
What is the difference between crawling and indexing?
Crawling is fetching a page. Indexing is analyzing it and storing it so it can appear in search results. A page can be crawled and still not be indexed.
How do I get Google to crawl my site?
Make sure your pages are linked internally, submit an XML sitemap in Google Search Console, and avoid blocking them in robots.txt. You can also request indexing for individual URLs with the URL Inspection tool.
How often does Google crawl a site?
It varies by site. Pages that are popular, important, or updated often tend to be crawled more frequently.
Want help putting this to work?
Tell us where your marketing stands today, and we'll show you what to do first.