On Sat, 2002-11-02 at 21:13, Bjoern Hoehrmann wrote: > Since the robots.txt resource says > > User-agent: * > Disallow: /check > > checklink should not even look at /check/refer. Checking /robots.txt is already in the TODO list [1] for checklink; this will most likely be implemented with LWP::RobotUA from the excellent libwww-perl package. [1] <http://www.w3.org/2000/07/checklink> (or htdocs/docs/checklink.html in validator CVS) -- \/ille Skyttä ville.skytta at iki.fiReceived on Saturday, 2 November 2002 14:40:30 UTC
This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 22:58:30 UTC