W3C home > Mailing lists > Public > www-validator@w3.org > April 2004

Re: [ANN] W3C Link Checker - version 3.9.2 - first standalone release

From: Matthew Wilson <matthew@mjwilson.demon.co.uk>
Date: Sun, 11 Apr 2004 19:12:10 +0100
Message-ID: <40798A7A.1030208@mjwilson.demon.co.uk>
To: www-validator@w3.org

> Does the Link Checker obey robots.txt?
> 
> 
>    "...the W3C has a link checker. It announces itself as
>    W3C-checklink/3.9.2 [3.17] libwww-perl/5.64. It does not respect
>    robots.txt"

It seems to me that changing LWP::UserAgent to LWP::RobotUA in 
'checklink' should be enough to fix this.

-- 
Matthew Wilson
Received on Sunday, 11 April 2004 14:12:13 UTC

This archive was generated by hypermail 2.3.1 : Wednesday, 7 January 2015 15:30:43 UTC