The link validator has a built-in one second delay between page fetches - I appreciate the reasons but when I am trying to check my own site it can take a while. As I don't care how fast you crawl my server at my request can I suggest the following: Check the root level for a file (maybe use robots.txt?) for a 'crawl-delay' value and if it's zero let the validator run at full speed. plus: Site admins can then use the same process to block the validator if they didn't request it. plus: minor stress testReceived on Monday, 2 January 2012 14:45:43 GMT
This archive was generated by hypermail 2.2.0+W3C-0.50 : Wednesday, 25 April 2012 12:14:50 GMT