Meta robots tag: noindex, nofollow, noarchive, noodp & noydir

Meta robots tag controls spiders from crawling and indexing your webpage or post. In this post, we will see all types of tags in detail. We can use more than one meta tags in a webpage but its not a good practice because it can cause conflicts. Hence it should occur only once in a webpage.

Default Meta robots tag

If you don’t specify any meta tag in a webpage then by default spider will crawl and index your webpage. you can say that it considers below meta tag by default.

<meta name="robots" content="index, follow"" />

<meta name="robots" content="none">

Meta robots noindex

It prevents page from being indexed. It means if you specify below meta tag for a page then that page would not come up in search results. For example: you wouldn’t want your author pages to come up in search results so to prevent it you can use below Meta robots tag in author pages.

<meta name="robots" content="noindex" />

Meta robots nofollow

This meta tag prevents spiders to follow any link on the page. If you are a newbie then it may be confusing to you. Let me explain you in detail. Whenever spider( Google bot) finds a link (internal or external) on a page, it lands on that webpage for crawling and indexing, to enforce spider for not following any link on the page, we can use this tag. To understand it better you can refer my post Dofollow vs nofollow.

<meta name="robots" content="nofollow" />

Meta robots noarchive

You must have seen in search results that whenever you hover on any result it will show a snapshot of that page at the right side of the Google search (see the screenshot below). That we call a cached copy of the webpage. To prevent a page from being cached we can specify below tag.

<meta name="robots" content="noarchive" />

Meta robots noodp

Prevents Search engine (Google, Yahoo and MSN)  to display its own description taken from its directory, instead of you meta description. Using below tag would ensure that Search engine would display your meta description below search results and it would be useful to improve your CTR.

<meta name="robots" content="noodp " />

Meta robots noydir

It is same as above NOODP tag, the only difference is that it would be applicable for Yahoo only.

<meta name="robots" content="noydir" />

Best Practice to use noodp and noydir: You can use the below tag for all of your webpages:

<meta name="robots" content="noodp, noydir" />   

OR (above tag and below two tags- both are same)

<meta name="robots" content="noodp " />

<meta name="robots" content="noydir" />

What if page has more than one Meta robots tag?

We should use it only once per page and its the best practice, but sometimes because of combination of SEO friendly themes and SEO plugins, we may have multiple instances of such tags. So for such cases lets discuss how spiders (or crawlers) would interpret it.

CASE 1: No conflict values in robots meta tag.

<meta name="robots" content="noindex" />

<meta name="robots" content="nofollow" />

Above two tags would be taken same as: <meta name="robots" content="noindex, nofollow" />

CASE 2: Conflicts value present in robots meta tag.

First instance: <meta name="robots" content="noindex" />

Second instance: <meta name="robots" content="index" />

In this case both values are conflicting as one tell to noindex and other is enforcing spider to index the page. For such cases crawlers considers first instance of conflicts.

So search engine bot would take it as <meta name="robots" content="noindex" /> because noindex encountres before than index one.

How to block specific search engine bot?

By specifying meta name as “robots”,you are actually blocking all search engine robots. However if you want to block specific search engine bot then you can specify bot name instead of “robots”

BOT Name:

Google: GOOGLEBOT

Yahoo: SLURP

MSN: MSNBOT

ASK: TEOMA

So, if you want to make your page nofollow for Google bot only then you can use below tag.

<meta name="Googlebot" content="noindex" />

Note: In meta name I have given “Googlebot” instead of “robots” to make it applicable for Google search engine bot only.

Reference

Meta robots tag: Google Webmaster blog

Recommended Posts

• On Page SEO Checklist
• Free LSI keywords Tools