W3C home > Mailing lists > Public > www-validator@w3.org > July 2000

Re: W3C® Link Checker:

From: Hugo Haas <hugo@w3.org>
Date: Tue, 25 Jul 2000 16:17:01 -0400
To: Lawrence, André <andre_lawrence@silanis.com>, "'www-validator@w3.org'" <www-validator@w3.org>
Message-ID: <20000725161700.A23303@w3.org>
On Mon, Jul 24, 2000, "Lawrence, André" wrote:
> Will you follow the exclude restriction of my robots.txt?

I thought about that a few times.

The link checker was developed in order to check links on the W3C Site.
Therefore only a few hits outside the W3C Site were made the program was
not crawling other sites.

Now that the service is used outside W3C, maybe it makes sense to
comply with the robots exclusion standard.

I have added that to the to-do list:


Received on Tuesday, 25 July 2000 16:17:07 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 22:58:17 UTC