Re: [ANN] W3C Link Checker - version 3.9.2 - first standalone release from Matthew Wilson on 2004-04-11 (www-validator@w3.org from April 2004)

From: Matthew Wilson <matthew@mjwilson.demon.co.uk>
Date: Sun, 11 Apr 2004 19:12:10 +0100
To: www-validator@w3.org
Message-ID: <40798A7A.1030208@mjwilson.demon.co.uk>

> Does the Link Checker obey robots.txt?
> 
> 
>    "...the W3C has a link checker. It announces itself as
>    W3C-checklink/3.9.2 [3.17] libwww-perl/5.64. It does not respect
>    robots.txt"

It seems to me that changing LWP::UserAgent to LWP::RobotUA in 
'checklink' should be enough to fix this.

-- 
Matthew Wilson

Received on Sunday, 11 April 2004 14:12:13 UTC