robots.txt

robots.txt is a plain text file sitting at the root of a website that tells automated crawlers which parts of the site they may request. It is the oldest piece of etiquette on the web and it works on the honour system: well behaved crawlers read it and comply, and anything determined to ignore it simply will.

In short

• A file at the root of your site telling crawlers where they may go.
• It controls crawling. It does not control what appears in results.
• Anybody can read yours, so nothing secret belongs in it.
• One stray line can remove an entire site from search.

The distinction everything else depends on

Crawling and indexing are two different actions, and robots.txt only touches the first. Crawling is a search engine fetching the page. Indexing is it deciding to store the page and show it to people. Blocking the fetch does not prevent the storing.

What happens in practice is that Google finds the address through a link on another site, is not allowed to look at it, and lists it anyway on the strength of that link. The result appears with no description, because there was nothing to read. The page owner blocked it specifically to keep it out of Google and it is in Google, with a blank entry that looks worse than the page would have.

Keeping something out of results is a job for a noindex tag on the page. Keeping a crawler out of a section of the site is a job for robots.txt. Using the wrong one is the most common mistake made with this file, and the second most common is using both at once.

Why using both fails

It sounds like belt and braces. It is closer to locking a note inside a box and posting the box. A noindex tag lives in the page. For Google to obey it, Google has to fetch the page and read it. Blocking the address in robots.txt prevents exactly that. So the instruction sits there unread, the block stays in force, and the address can persist in results for a long time with nothing you can do from the file.

The correct sequence, when a page is already blocked and you want it gone, is to remove the block, let the page be crawled, let the noindex be read and acted on, and only then decide whether the block is still wanted. Counterintuitive, and it is the order that works.

What the file can contain

Google reads four fields and ignores everything else.

• User-agent. Which crawler the following rules apply to. An asterisk means all of them.
• Disallow. A path the crawler should not request.
• Allow. An exception carved out of a Disallow, for letting one thing through inside a blocked folder.
• Sitemap. The full address of your XML sitemap. Not a rule, just a signpost, and the usual reason a small site has the file at all.

Crawl-delay is the notable absence. Other search engines honour it and Google does not, so a site expecting it to throttle Googlebot is not getting what it thinks. Google also caps how much of the file it will read at 500 kibibytes, which no normal site will approach.

Paths are case-sensitive. A rule written for a lowercase folder will not cover the same folder capitalised, and on servers that accept both spellings that gap is easy to miss.

Words you will hear

• Googlebot. The name Google’s crawler identifies itself by, and the value you would write after User-agent to address it alone.
• Wildcard. An asterisk standing in for any run of characters. Its partner is the dollar sign, which anchors a rule to the end of an address, and forgetting that one is how a rule meant for a file type ends up matching far more than intended.
• noindex. A tag on the page itself asking that it be kept out of results, and for files that are not web pages, a header doing the same job. The tool robots.txt is repeatedly mistaken for.
• Crawl efficiency. How well a site spends the attention a search engine is willing to give it. Sometimes called crawl budget, and only a real concern once a site is large.
• Indexed though blocked. The Search Console status describing exactly the failure at the top of this page.

The line that empties a site

There is one mistake here that outranks all others in consequence.

A site under development is normally blocked from everything, and the way that is done is a Disallow rule with a single slash after it, which means the whole site. When the site goes live, that line has to come out. Sometimes it does not.

The site launches, looks perfect, and quietly instructs every search engine to stay away from all of it. Nothing visibly breaks. Traffic simply never arrives, and because a new site has no history, nobody notices the absence for weeks.

Load your own domain followed by /robots.txt right now and read what is there. It takes ten seconds. If a Disallow line is followed by nothing but a slash, that is the problem and it has been the problem since launch.

What it is legitimately good for

Set against those warnings, the file does earn its place on larger sites. A shop with filters can generate an effectively unlimited number of addresses, every combination of colour and size and sort order producing another page of the same products. None of them needs crawling. Blocking that pattern keeps a search engine spending its attention on the pages that matter, which is the crawl efficiency argument described on the technical SEO entry.

Internal search result pages are the other clear case. They are generated on demand, there are as many of them as there are things to search for, and none is worth storing.

Most small businesses, though, need to do two things with this file and nothing more: check that it is not blocking the site, and make sure it names the sitemap. Both are one line, and both are the sort of thing that gets set up once during a build and then never looked at again until something has gone wrong for a month.