Blog

Does my website need a sitemap, a robots.txt file or a noindex tag?

Your web designer has mentioned a sitemap, or a report has flagged your robots.txt, and you are not sure what either one does. The names sound alike, and the jobs are different. This post takes them one at a time, says what Google says each one is for, and shows the mix-up that can leave a page you meant to hide in Google.

What does each one do?

Each answers a different question. A sitemap says which pages you think matter. A robots.txt file says where crawlers may go. A noindex rule says which pages should stay out of the results. Here is each one in Google's terms:

  • XML sitemap: a file that lists the pages and files you consider important, and can say when each one was last updated (Google Search Central).
  • robots.txt: a file that tells crawlers which URLs they may access, mainly so they do not overload your site (Google Search Central).
  • noindex: a rule set as a meta tag on the page or as an HTTP response header, which makes Google drop that page from its search results (Google Search Central).

Does a small site need a sitemap?

It depends on how easy your pages are to find. Google says a site whose pages are well linked can usually be discovered without a sitemap, and that large sites and new sites with few links from elsewhere benefit most (Google Search Central). A site that went live last month, with nobody linking to it yet, is the kind Google says benefits most.

Look before you make one. Google says site builders often publish a sitemap automatically (Google Search Central), so yours may already exist. Check your builder's settings, or ask whoever built the site.

Read it as a list, never as an order. Google says a sitemap helps it discover pages but does not mean every URL in it will be crawled or indexed (Google Search Central).

The free scan looks for one. T7 fires when there is no readable XML sitemap, or when more than a tenth of the sitemap URLs it checks, at least 5 of them, are not live and indexable. T8 lists sitemap URLs that no page it read links to, each a possible orphan page. The scan only knows the pages it read, so one it did not reach may already link to it (how the scan works).

Do you need a robots.txt file at all?

Write one when you have something to tell crawlers about which URLs they may access, the job Google gives the file (Google Search Central).

You may have one already. Google says site builders often manage the file themselves (Google Search Central), so there may be lines in yours that nobody at your business chose. Ask whoever runs your site to show it to you, and read each line.

The line to look for is one that turns Google away from pages you want found. A robots.txt that disallows Googlebot on key URLs is a high severity check in our free scan (T1), and the scan reads the file before anything else.

Can robots.txt keep a page out of Google?

No. Google says robots.txt is not a way to keep a page out of its results, and that a page the file blocks can still be indexed if other pages link to it (Google Search Central). Blocking a page and removing it from the results are two separate things.

For a page that has to stay out, Google points to noindex or a password (Google Search Central). Use a password for anything private, such as a price sheet made for one customer or a page for your staff.

How do you use noindex without undoing it?

Put the rule on the page and leave the page open to crawlers. Google says a noindex rule works only if robots.txt does not block the page, and that a noindex rule written inside robots.txt is not supported (Google Search Central). So never pair a block with noindex on the same page. For a page that should stay out of the results, use noindex and leave the page unblocked.

Good candidates are pages with nothing for a searcher, such as the thank-you page after a form, a test page, or a page for an old offer you keep for existing customers. Then make sure it never lands where it should not. Our free scan reports noindex on the homepage or a main navigation page as a high severity finding (T2), one of the technical SEO problems it looks for.

What if one page answers at more than one address?

That is where a fourth tool comes in, the canonical tag. When the same page sits at several URLs, which is one kind of duplicate content, Google treats one of them as canonical. Google says redirects and rel="canonical" are strong signals for which one, and that listing a URL in a sitemap is a weak one (Google Search Central).

None is required: given no preference, Google chooses a version itself, and reasons to state one include choosing the URL shown in results and consolidating signals such as links (Google Search Central). Use a 301 redirect for an old address nobody should land on any more, and a canonical tag for a copy that has to stay reachable, such as a page with a tracking code on the end.

Keep your signals in step. List the canonical address in your sitemap, not the copies. The free scan flags a canonical tag that points to another host, a redirect or an error page (T3).

Checking all four, counted

Here is the time a check of your robots.txt, sitemap, noindex rules and canonical tags takes on a small site. The page counts and minutes are made-up round numbers for the example, so put in your own.

Read your robots.txt, line by line
10 minutes
Compare your sitemap with your menu
12 pages at 1 minute each: 12 minutes
Pages meant to carry noindex: a thank-you page, a test page, an old offer
3 pages at 4 minutes each: 12 minutes
The homepage and 5 menu pages, to confirm none carries noindex
6 pages at 2 minutes each: 12 minutes
Pages that answer at a second address, to check the canonical tag
2 pages at 5 minutes each: 10 minutes
One check
10 + 12 + 12 + 12 + 10 = 56 minutes
Checks a year: once, and once after any redesign
2
Total
2 checks at 56 minutes: 112 minutes, just under 2 hours a year

Just under 2 hours a year. Keep the check after a redesign. Google says site builders often publish the sitemap (Google Search Central) and often manage robots.txt themselves (Google Search Central), so a new builder can change both at once.

Fixing what you find is not in the count. Note each page that fails, and hand the list to whoever edits your site.

Round numbers for the arithmetic, not a benchmark and not a forecast. Put in your own.

Questions

Can I put a noindex line in my robots.txt file?

No. Google says a noindex rule in robots.txt is not supported (Google Search Central). Set it as a meta tag on the page, or send it as an HTTP response header, and leave that page unblocked.

Which pages belong in my sitemap?

The ones you want found. Google describes a sitemap as a list of the pages and files you consider important (Google Search Central), and listing a URL there is a weak signal for which of several duplicates you prefer (Google Search Central). Leave out pages marked noindex and old addresses that redirect.

If I take a page down, should I block it in robots.txt?

No. Google says a page blocked by robots.txt can still be indexed if other pages link to it (Google Search Central). If the page has a replacement, send its old address there with a 301 redirect. If it must stay up but out of the results, use noindex (Google Search Central).

Related reading

Posts

Guides

Sources