W3C home > Mailing lists > Public > public-qa-dev@w3.org > June 2004

Re: looking for feedback - robots implementation for link checker

From: olivier Thereaux <ot@w3.org>
Date: Wed, 30 Jun 2004 08:34:37 +0900
Message-Id: <E2A18BE3-CA24-11D8-933A-000A95E54002@w3.org>
Cc: QA Dev <public-qa-dev@w3.org>
To: Susan Lesch <lesch@w3.org>
Susan, thanks for your feedback.
I am copying the developers' list in my reply, hoping it is fine with  
you.

On Jun 30, 2004, at 7:47, Susan Lesch wrote:
>  With default settings, for http://www.w3.org/ that checked "1  
> document in 160.4 seconds" vs. 8.5 seconds for ,checklink.

Indeed, you are right.

My understanding of the code changes between the current :80 and :8001
http://dev.w3.org/cvsweb/perl/modules/W3C/LinkChecker/bin/ 
checklink.diff?r1=3.58&r2=3.17&f=h
is that the sleep_time parameter is now used not only as a delay  
between checking documents in recursive mode, but also as a delay  
between HEAD requests checking links from a given document.

That (as well as the robots implemntation, BTW)  was, IIRC, decided as  
a mean to respond to critics saying that the link checker was a  
"misbehaving" agent. OTOH, obviously, it does make it very slow for  
documents with a lot of links. We should try to find a good tradeoff (I  
think the idea of having a specific intra-w3c instance without such  
limitations was one idea) but this is a difficult balance to achieve.

Thanks
-- 
olivier

Received on Tuesday, 29 June 2004 19:34:53 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Thursday, 19 August 2010 18:12:44 GMT