- From: Ville Skyttä <ville.skytta@iki.fi>
- Date: 02 Nov 2002 21:45:44 +0200
- To: Bjoern Hoehrmann <derhoermi@gmx.net>
- Cc: Terje Bless <link@pobox.com>, W3C Validator <www-validator@w3.org>
On Sat, 2002-11-02 at 21:13, Bjoern Hoehrmann wrote:
> Since the robots.txt resource says
>
> User-agent: *
> Disallow: /check
>
> checklink should not even look at /check/refer.
Checking /robots.txt is already in the TODO list [1] for checklink; this
will most likely be implemented with LWP::RobotUA from the excellent
libwww-perl package.
[1] <http://www.w3.org/2000/07/checklink>
(or htdocs/docs/checklink.html in validator CVS)
--
\/ille Skyttä
ville.skytta at iki.fi
Received on Saturday, 2 November 2002 14:40:30 UTC