W3C home > Mailing lists > Public > www-validator@w3.org > December 2002

Re: Unwanted robot accesses from your site

From: Olivier Thereaux <ot@w3.org>
Date: Wed, 25 Dec 2002 16:05:49 +0900
Cc: www-validator@w3.org
To: Bjoern Hoehrmann <derhoermi@gmx.net>
Message-Id: <4C1F6982-17D7-11D7-B932-000393BAB03A@w3.org>

Hi Bjoern.

On Wednesday, Dec 25, 2002, at 13:48 Asia/Tokyo, Bjoern Hoehrmann wrote:
> Why doesn't checklink qualify as a robot?

My own definition of a robot is that it retrieves some data (the 
documents) or metadata (indexing). I may be wrong.

In any case I don't think checklink, even in recursive mode, should 
follow the robots directives (noindex is irrelevant, and nofollow would 
make it useless...). I'm interested to hear opposite arguments, though.

Received on Wednesday, 25 December 2002 02:05:59 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 22:58:31 UTC