W3C home > Mailing lists > Public > www-validator@w3.org > January 2012

Link validator crawl speed

From: Paul <paul@ianlynam.com>
Date: Mon, 2 Jan 2012 12:28:09 +0900
Message-Id: <E169AC97-BC9D-4F62-B41C-4D8CB48C6CA9@ianlynam.com>
To: www-validator@w3.org
The link validator has a built-in one second delay between page fetches - I appreciate the reasons but when I am trying to check my own site it can take a while. 

As I don't care how fast you crawl my server at my request can I suggest the following:

Check the root level for a file (maybe use robots.txt?) for a 'crawl-delay' value and if it's zero let the validator run at full speed. 

plus: Site admins can then use the same process to block the validator if they didn't request it.
plus: minor stress test
Received on Monday, 2 January 2012 14:45:43 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 22:59:25 UTC