> Does the Link Checker obey robots.txt? > > > "...the W3C has a link checker. It announces itself as > W3C-checklink/3.9.2 [3.17] libwww-perl/5.64. It does not respect > robots.txt" It seems to me that changing LWP::UserAgent to LWP::RobotUA in 'checklink' should be enough to fix this. -- Matthew WilsonReceived on Sunday, 11 April 2004 14:12:13 GMT
This archive was generated by hypermail 2.2.0+W3C-0.50 : Monday, 7 December 2009 10:57:13 GMT