Re: [ANN] W3C Link Checker - version 3.9.2 - first standalone release from Pete Prodoehl on 2004-04-02 (www-validator@w3.org from April 2004)

From: Pete Prodoehl <pete.prodoehl@cygnusinteractive.com>
Date: Fri, 02 Apr 2004 13:40:57 -0600
To: www-validator <www-validator@w3.org>
Message-ID: <406DC1C9.1020906@cygnusinteractive.com>

olivier Thereaux wrote:
> -=====================================================================-
> -     W3C Link Checker - version 3.9.2 - first standalone release     -
> -=====================================================================-
> 

Does the Link Checker obey robots.txt?

   "...the W3C has a link checker. It announces itself as
   W3C-checklink/3.9.2 [3.17] libwww-perl/5.64. It does not respect
   robots.txt and is therefore banned in .htaccess with the rule"

    RewriteCond %{HTTP_USER_AGENT} ^W3C-checklink
    RewriteRule .* - [F,L]

   http://diveintomark.org/archives/2004/04/02/ten-words

Pete

Received on Friday, 2 April 2004 14:41:45 UTC