- From: Nick Kew <nick@webthing.com>
- Date: Sun, 4 Apr 2004 22:33:01 +0100 (BST)
- To: www-validator@w3.org
> > I don't know why htmlhelp.com blocks the link checker, but I wouldn't be > > surprised if it was something to do with the way it (the link checker) > > ignores the robots exclusion standard. > > Yes, that's why. I banned W3C-checklink after it overloaded our server, > preventing legitimate users from accessing it. W3C-checklink had been > spidering resources that were disallowed in robots.txt. A while ago I had email from Liam drawing my attention to the fact that I'd introduced a similar bug into htmlhelp's own link checker. I've since fixed that, and round tuits permitting, I'll take a look at the same issue in checklink. It can be very easy to overlook bugs when you don't have someone complaining about them at a convenient time. -- Nick Kew Nick's manifesto: http://www.htmlhelp.com/~nick/
Received on Sunday, 4 April 2004 17:33:33 UTC