Re: Benefits of :<META NAME="ROBOTS" CONTENT="NOINDEX,FOLLOW">

To: Don <dledarney@man.net>, www-disw-request@w3.org
Subject: Re: Benefits of :<META NAME="ROBOTS" CONTENT="NOINDEX,FOLLOW">
From: Martijn Koster <mak@excitecorp.com>
Date: Wed, 23 Jun 1999 19:22:31 +0100
From mak@mantra.greenhills.co.uk Wed Jun 23 14: 22:15 1999
In-Reply-To: <000801bebdac$aadd3180$e830c8cd@don>; from Don on Wed, Jun 23, 1999 at 12:14:49PM -0700
References: <000801bebdac$aadd3180$e830c8cd@don>

> <META NAME="ROBOTS" CONTENT="NOINDEX,FOLLOW">

FOLLOW refers to extracting links from a page, INDEX refers to
extracting content and indexing it. For example, if you have

 page1.html:

   <a href="page2.html">here is another page</a> for you

When a robot sees "NOINDEX,FOLLOW", it ignores the text "here is another
page for you", but does go and retrieve page2.html.


> The info. I've read doesn't describe the benefits of the "noindex, follow".

See http://info.webcrawler.com/mak/projects/robots/exclusion.html#meta

-- Martijn Koster, m.koster@pobox.com

References:

Benefits of :<META NAME="ROBOTS" CONTENT="NOINDEX,FOLLOW">

From: "Don" <dledarney@man.net>

Prev: MAIL DELIVERY FAILURE
Next: Returned mail: User unknown
Index: Message index of www-disw@w3.org mailing list
Thread: Thread index of www-disw@w3.org mailing list