What is an AI crawler?
An AI crawler is an automated program that visits web pages to collect content for an AI product. Depending on the crawler, that content may be used to train a model, to build a search index, or to answer a user's question.
Last reviewed .
What AI crawlers do
AI companies run several kinds of bots. Some gather content to train models, some build an index that powers search features, and some fetch a page live when a user asks about it.
Each purpose has different consequences for you. Blocking a training bot is a different choice from blocking a bot that lets your pages appear in AI answers.
How to manage AI crawlers
Most reputable crawlers state a name and follow the rules in your robots.txt file. You can allow or disallow each one separately, and your hosting provider or firewall may offer additional controls.
Review the current documentation from each AI company before you decide, because crawler names and purposes change. Also confirm that important pages aren't hidden behind scripts or logins that a crawler can't pass.
Common questions
Should I block AI crawlers?
It depends on your goals. Blocking may limit how your content is used for training, but it can also keep your pages out of AI answers. Decide by crawler purpose, not with a blanket rule.
How do I block an AI crawler?
Add a rule for the crawler's user agent name to your robots.txt file. Well-behaved bots follow it, but it isn't enforcement, so use server or firewall rules if you need a hard block.
How can I tell if AI crawlers visit my site?
Check your server logs or your hosting or CDN analytics for known AI user agent names. Standard analytics tools usually don't show bot visits.
Is an AI crawler the same as Googlebot?
Sometimes the same crawler supports both search and AI features. Googlebot crawls for Google Search, including AI Overviews and AI Mode. To keep a site out of those AI features without blocking Googlebot, use the Search generative AI setting in Search Console. Google-Extended controls use for model training. Other providers may use separate crawlers for search indexing, training, and user-requested retrieval.
Want help putting this to work?
Tell us where your marketing stands today, and we'll show you what to do first.