W3C home > Mailing lists > Public > www-validator@w3.org > April 2004

Re: [ANN] W3C Link Checker - version 3.9.2 - first standalone release

From: Pete Prodoehl <pete.prodoehl@cygnusinteractive.com>
Date: Fri, 02 Apr 2004 13:40:57 -0600
Message-ID: <406DC1C9.1020906@cygnusinteractive.com>
To: www-validator <www-validator@w3.org> 

olivier Thereaux wrote:
> -=====================================================================-
> -     W3C Link Checker - version 3.9.2 - first standalone release     -
> -=====================================================================-
> 


Does the Link Checker obey robots.txt?


   "...the W3C has a link checker. It announces itself as
   W3C-checklink/3.9.2 [3.17] libwww-perl/5.64. It does not respect
   robots.txt and is therefore banned in .htaccess with the rule"

    RewriteCond %{HTTP_USER_AGENT} ^W3C-checklink
    RewriteRule .* - [F,L]


   http://diveintomark.org/archives/2004/04/02/ten-words


Pete
Received on Friday, 2 April 2004 14:41:45 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Monday, 7 December 2009 10:57:12 GMT