- From: Liam Quinn <liam@htmlhelp.com>
- Date: Sun, 4 Apr 2004 11:17:48 -0400 (EDT)
- To: David Dorward <david@dorward.me.uk>
- Cc: MichaelJennings <mikejenn@fiam.net>, <www-validator@w3.org>
On Sun, 4 Apr 2004, David Dorward wrote: > On Sun, 2004-04-04 at 08:46, MichaelJennings wrote: > > http://www.htmlhelp.com/ > > HTTP Code returned: 403 > > HTTP Message: Forbidden > > Actually, I think if you try the URL you'll find it is > > not only permitted, but pretty good competition. > > Really? > > david@cyberman david $ pavuk -identity "W3C-checklink/3.9.2 [3.17] > libwww-perl/5.64" http://www.htmlhelp.com/ > > http://www.htmlhelp.com/ URL[ 1]: 1(0) of 1 > http://www.htmlhelp.com/ > download: ERROR: forbidden HTTP request > > Certainly seems to be forbidden to me. > > I don't know why htmlhelp.com blocks the link checker, but I wouldn't be > surprised if it was something to do with the way it (the link checker) > ignores the robots exclusion standard. Yes, that's why. I banned W3C-checklink after it overloaded our server, preventing legitimate users from accessing it. W3C-checklink had been spidering resources that were disallowed in robots.txt. -- Liam Quinn
Received on Sunday, 4 April 2004 11:17:50 UTC