Re: checklink: htmlhelp is forbidden

On Sun, 4 Apr 2004, David Dorward wrote:

> On Sun, 2004-04-04 at 08:46, MichaelJennings wrote:
> > http://www.htmlhelp.com/
> > HTTP Code returned: 403
> > HTTP Message: Forbidden
> > Actually, I think if you try the URL you'll find it is
> > not only permitted, but pretty good competition.
> 
> Really? 
> 
> david@cyberman david $ pavuk -identity "W3C-checklink/3.9.2 [3.17]
> libwww-perl/5.64" http://www.htmlhelp.com/
> 
> http://www.htmlhelp.com/ URL[ 1]:     1(0) of     1 
> http://www.htmlhelp.com/
> download: ERROR: forbidden HTTP request
> 
> Certainly seems to be forbidden to me.
> 
> I don't know why htmlhelp.com blocks the link checker, but I wouldn't be
> surprised if it was something to do with the way it (the link checker)
> ignores the robots exclusion standard.

Yes, that's why.  I banned W3C-checklink after it overloaded our server,
preventing legitimate users from accessing it.  W3C-checklink had been
spidering resources that were disallowed in robots.txt.

-- 
Liam Quinn