What is robots.txt?
Robots.txt is a plain text file at the root of your website that tells search engine crawlers which pages or folders they may or may not request. It manages crawling, but it doesn't keep pages out of search results.
Last reviewed .
What robots.txt does
The file contains rules for crawlers, such as which paths to avoid. Well-behaved crawlers, including Googlebot, check it before crawling a site.
It can also point crawlers to your XML sitemap.
What robots.txt can't do
Blocking a page from crawling doesn't remove it from search. If other sites link to it, the URL can still be indexed without its content. To keep a page out of results, use a noindex tag and let it be crawled so the tag can be read.
It is also not a security measure. The file is public, and not all bots obey it.
Common mistakes
Accidentally blocking the whole site, or blocking CSS and JavaScript files that pages need to render, are frequent errors. Test changes before they go live and review the file after site launches.
Common questions
Where does the robots.txt file go?
It must sit at the root of the domain, for example example.com/robots.txt. Crawlers won't look for it elsewhere.
Does robots.txt hide pages from Google?
No. It stops crawling, not indexing. Use a noindex tag or password protection for pages that shouldn't appear in search.
How do I check my robots.txt?
Open it in a browser at your domain's /robots.txt address. Google Search Console also includes a robots.txt report showing what Google found.
Can robots.txt block AI crawlers?
Many AI crawlers publish user agent names and say they follow robots.txt rules, so you can disallow them there. Compliance depends on the crawler.
Want help putting this to work?
Tell us where your marketing stands today, and we'll show you what to do first.