Crawler (bot), explained
Google calls crawler a general term for any program that automatically finds and scans websites. Googlebot is the main one Google uses for Search. What crawlers read is how a search engine or an AI tool learns what is on your site.
AI companies run more than one crawler, and each one is a separate switch. OpenAI says OAI-SearchBot is for search, so blocking it keeps a site out of ChatGPT search answers. GPTBot collects pages that may be used to train its models. A site can allow one and block the other.
What does robots.txt do for crawlers?
It is a text file at the root of a site that says which crawlers may visit which paths. The rules are a request, not a lock: the Robots Exclusion Protocol standard says crawlers are asked to follow them, and that the rules are not a form of access control. Google says its common crawlers always respect robots.txt for automatic crawls.
Should a business block AI crawlers?
It depends on what you want. Blocking a search crawler can keep you out of that tool’s answers. Blocking a training crawler only says your pages should not be used to train models. Decide one crawler at a time, not all at once.
Questions
Is a crawler the same as a bot?
A crawler is one kind of bot. Google says crawlers are sometimes called robots or spiders. But not every bot is a crawler: some fill in forms or send spam.
What is Googlebot?
Googlebot is the main crawler Google uses for Google Search. Google says it always respects robots.txt rules for automatic crawls.
What is the difference between OAI-SearchBot and GPTBot?
OAI-SearchBot is how OpenAI finds sites for ChatGPT search. GPTBot collects pages that may be used to train OpenAI’s models. Each has its own robots.txt setting.
Where we use this term
Sources
- Overview of Google crawlers and fetchers (user agents). Google Search Central.
- RFC 9309: Robots Exclusion Protocol. IETF. .
- Overview of OpenAI Crawlers. OpenAI.
Definition checked against these sources and our own pages on .