checklink: Offer a menu of time-delays between hits to server

Hello W3C Link Checker team:

Your current Link Checker documentation says, "Both the command line  
version and the online one sleep at least one second between requests  
to each server to avoid abuses and target server congestion."
However, this is causing problems. The time delay is too short, at  
least for the web host I use. Specifically, after the Link Checker has  
been running awhile, more and more of my links get returned with a 500  
code, saying "server closed connection without sending any data back."

In your bugs list, I noticed that Bug 918 has a very practical  
suggestion: Make the delay between hits to the server configurable. ( https://www.w3.org/Bugs/Public/show_bug.cgi?id=918 
#c0 )

I would like to see the W3C Link Checker offer a menu of time delays  
such as:
1 second
2 seconds
5 seconds
and so on

The user can then experiment on their own, choosing different delays  
until the system works well on their own web host.

Georgena Sil

Received on Monday, 17 September 2012 10:16:34 UTC