W3C home > Mailing lists > Public > www-validator@w3.org > April 2004

Re: [ANN] W3C Link Checker - version 3.9.2 - first standalone release

From: Matthew Wilson <matthew@mjwilson.demon.co.uk>
Date: Sun, 11 Apr 2004 19:12:10 +0100
Message-ID: <40798A7A.1030208@mjwilson.demon.co.uk>
To: www-validator@w3.org

> Does the Link Checker obey robots.txt?
> 
> 
>    "...the W3C has a link checker. It announces itself as
>    W3C-checklink/3.9.2 [3.17] libwww-perl/5.64. It does not respect
>    robots.txt"

It seems to me that changing LWP::UserAgent to LWP::RobotUA in 
'checklink' should be enough to fix this.

-- 
Matthew Wilson
Received on Sunday, 11 April 2004 14:12:13 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Wednesday, 25 April 2012 12:14:13 GMT