Link validator crawl speed

The link validator has a built-in one second delay between page fetches - I appreciate the reasons but when I am trying to check my own site it can take a while. 

As I don't care how fast you crawl my server at my request can I suggest the following:

Check the root level for a file (maybe use robots.txt?) for a 'crawl-delay' value and if it's zero let the validator run at full speed. 

plus: Site admins can then use the same process to block the validator if they didn't request it.
plus: minor stress test

Received on Monday, 2 January 2012 14:45:43 UTC