W3C home > Mailing lists > Public > www-validator@w3.org > December 2002

Re: Unwanted robot accesses from your site

From: Olivier Thereaux <ot@w3.org>
Date: Wed, 25 Dec 2002 16:05:49 +0900
Cc: www-validator@w3.org
To: Bjoern Hoehrmann <derhoermi@gmx.net>
Message-Id: <4C1F6982-17D7-11D7-B932-000393BAB03A@w3.org>

Hi Bjoern.

On Wednesday, Dec 25, 2002, at 13:48 Asia/Tokyo, Bjoern Hoehrmann wrote:
> Why doesn't checklink qualify as a robot?

My own definition of a robot is that it retrieves some data (the 
documents) or metadata (indexing). I may be wrong.

In any case I don't think checklink, even in recursive mode, should 
follow the robots directives (noindex is irrelevant, and nofollow would 
make it useless...). I'm interested to hear opposite arguments, though.

-- 
Olivier
Received on Wednesday, 25 December 2002 02:05:59 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Wednesday, 25 April 2012 12:14:05 GMT