XML sitemap

An XML sitemap is a file on your website that lists the addresses you want a search engine to know about. It is written for machines rather than people, which is why opening it in a browser produces a wall of tags, and its only job is discovery: telling a crawler that these pages exist and roughly when each one last changed.

In short

• A machine-readable list of the pages you want indexed.
• It helps pages get found. It does not help them rank.
• Your platform almost certainly generates one already.
• Google still decides separately whether to keep each page.

What is inside one

Each entry is an address and, usually, a date. That is close to the whole format.

There are two optional fields, priority and change frequency, that once looked useful and are now ignored by Google, because almost everybody set every page to the maximum and the values stopped meaning anything. The last-modified date is still read, on one condition: it has to be true. A system that writes today’s date against every page every night is not providing information, and an engine that notices will stop trusting the field.

Large sites split the list. The limit is fifty thousand addresses or fifty megabytes uncompressed per file, and beyond that you use a sitemap index, which is a small file whose entries are other sitemap files. Most platforms cross that bridge for you.

What it is actually for

Search engines find pages mainly by following links. A page linked from your navigation, from a category listing, and from three other articles will be found without any help at all. The sitemap matters in the gaps. A brand new site has nothing pointing at it from anywhere, so there are no links to follow. A large shop has product pages sitting several clicks deep behind filters. A page published today would be found eventually through your own menu, and the sitemap tends to get it looked at sooner.

None of that is ranking work. It decides whether a page is in the running.

What it cannot do

This is where expectations usually go wrong, so it is worth being blunt about the limits.

• It does not make a page rank. Discovery and merit are separate questions.
• It does not guarantee indexing. Google evaluates each page on its own and routinely declines to keep ones it judges thin or duplicated.
• It does not rescue a page nothing links to. Being crawled and being treated as important are not the same thing.
• It does not override a block. A page marked noindex stays out of results. A page disallowed in robots.txt stays uncrawled, which is not the same thing, and its address can still be listed. Either way the sitemap is an invitation rather than an instruction.

The last point causes real confusion. Listing a page in the sitemap while telling the engine elsewhere to ignore it is a contradiction, and Search Console will flag it rather than silently pick a side.

Words you will hear

• sitemap.xml. The usual filename, sitting at the root of the domain.
• Sitemap index. One file listing several sitemap files, used once a site passes the size limit.
• robots.txt. A separate file at the root that tells crawlers where not to go. It commonly names the sitemap location on one line.
• lastmod. The date a page last meaningfully changed.
• Canonical. The version of a page you have declared as the real one. Only canonical addresses belong in a sitemap.
• HTML sitemap. A normal page listing your pages, written for visitors. A different thing with a confusingly similar name.
• Indexed. Stored by the search engine and eligible to appear. The status a sitemap entry is hoping to reach, not one it confers.

Getting it in front of Google

Two ways, and doing both is fine. Submit the address once in Google Search Console, under Sitemaps, which also gives you a report of how many entries were read and when the file was last fetched. Separately, name it in your robots.txt file on a line beginning with Sitemap, which every major engine reads, Google included. Neither needs repeating. Once a sitemap is known, it gets refetched on its own schedule.

The report is the part worth returning to. It is the quickest place to see that the file is being read at all, and the Pages report beside it will show you which submitted addresses were not kept, which is a far more interesting question than whether the sitemap exists.

The mistake worth avoiding

The common failure is not a missing sitemap. It is a sitemap treated as the thing that gets pages found, while the site itself links to nothing.

Think about what it means for a page to be reachable only through the sitemap. You have told a search engine the page exists, and every other signal on your site says it does not matter enough to link to. The file cannot argue with that, and it was never designed to. Internal links are where that gets fixed.

So the useful exercise is not checking that the sitemap exists. It is listing the addresses in it, listing the addresses your own pages point at, and looking at the difference. An afternoon, and usually uncomfortable.

Do that once a year, keep an eye on the Sitemaps report in between, and the file can go back to being what it should be: one of the quieter items in technical SEO that nobody has to think about again.