W3C home > Mailing lists > Public > www-validator@w3.org > April 2004

Re: checklink: htmlhelp is forbidden

From: Liam Quinn <liam@htmlhelp.com>
Date: Sun, 4 Apr 2004 11:17:48 -0400 (EDT)
To: David Dorward <david@dorward.me.uk>
Cc: MichaelJennings <mikejenn@fiam.net>, <www-validator@w3.org>
Message-ID: <Pine.LNX.4.44.0404041111480.30170-100000@localhost.localdomain>

On Sun, 4 Apr 2004, David Dorward wrote:

> On Sun, 2004-04-04 at 08:46, MichaelJennings wrote:
> > http://www.htmlhelp.com/
> > HTTP Code returned: 403
> > HTTP Message: Forbidden
> > Actually, I think if you try the URL you'll find it is
> > not only permitted, but pretty good competition.
> 
> Really? 
> 
> david@cyberman david $ pavuk -identity "W3C-checklink/3.9.2 [3.17]
> libwww-perl/5.64" http://www.htmlhelp.com/
> 
> http://www.htmlhelp.com/ URL[ 1]:     1(0) of     1 
> http://www.htmlhelp.com/
> download: ERROR: forbidden HTTP request
> 
> Certainly seems to be forbidden to me.
> 
> I don't know why htmlhelp.com blocks the link checker, but I wouldn't be
> surprised if it was something to do with the way it (the link checker)
> ignores the robots exclusion standard.

Yes, that's why.  I banned W3C-checklink after it overloaded our server,
preventing legitimate users from accessing it.  W3C-checklink had been
spidering resources that were disallowed in robots.txt.

-- 
Liam Quinn
Received on Sunday, 4 April 2004 11:17:50 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Wednesday, 25 April 2012 12:14:13 GMT