- From: Hugo Haas <hugo@w3.org>
- Date: Tue, 25 Jul 2000 16:17:01 -0400
- To: Lawrence, André <andre_lawrence@silanis.com>, "'www-validator@w3.org'" <www-validator@w3.org>
On Mon, Jul 24, 2000, "Lawrence, André" wrote: > Will you follow the exclude restriction of my robots.txt? I thought about that a few times. The link checker was developed in order to check links on the W3C Site. Therefore only a few hits outside the W3C Site were made the program was not crawling other sites. Now that the service is used outside W3C, maybe it makes sense to comply with the robots exclusion standard. I have added that to the to-do list: http://www.w3.org/2000/07/checklink#what Hugo
Received on Tuesday, 25 July 2000 16:17:07 UTC