robots.txt, explained
Google’s guide says the file tells crawlers which URLs they can access. The main use is to keep a site from being flooded with requests. It is not a way to keep a page out of Google. To keep a page out, use a noindex rule or a password.
A blocked page can still show as a bare link if another site points at it, with no description. Some crawlers ignore the file. Do not hide secrets in it and hope. On this site the file is a traffic note. We name which crawler a block is for before anything is blocked.
Does robots.txt keep a page out of search?
Not by itself. Google says a disallowed URL can still be indexed if other sites link to it. The result may show the address without a snippet. Hiding the page takes a different tool.
Should I block AI crawlers in robots.txt?
Only when you can tell the crawlers apart. A block aimed at a training crawler is a different switch from a search crawler. If they cannot be told apart, we do not block. A block on the search crawler can also stop the page from being read for answers.
Questions
What is robots.txt in simple terms?
It is a plain text file at the root of a site that tells crawlers which URLs they may fetch and which to skip.
Will every crawler obey robots.txt?
No. Google says respectable crawlers follow it, and others might not. It cannot enforce a rule. A password is the tool for something that must stay private.
Is robots.txt the same as llms.txt?
No. robots.txt talks to crawlers about which URLs to fetch. llms.txt is a proposed list of pages you want a model to read. Google says it does not use llms.txt.
Where we use this term
Sources
- Introduction to robots.txt. Google Search Central. .
Definition checked against these sources and our own pages on .